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ABSTRACT 

This is a report of a &-day research training session 
held in New York City from January 30 to February 3, 1971 under the 
sponsorship of the American Educational Research Association (AERA) 
with support from the U.S. Office of Education. The piirpose of the 
training session was to develop and improve research competencies of 
individuals engaged in counseling, counselor education, and related 
research. The program purpose was implemented in two primary aims: 
developing the participants' understanding of systems research 
principles and concepts, and helping participants acquire proficiency 
in systems research skills and techniques. Twenty^two trainees were 
selected to participate in the training session. Program activities 
included didactic instruction, individualized and group activities, 
and supervised practice. Immc-idiate evaluation using internal 
criterion measures revealed attainment of program objective at or 
near performance level. Follow-up will be conducted to asisess 
long-range results. Progran evaluation indicated satisfacicion witih 
all aspects of organization and administration of the training 
session, with the exception of allocation of time and physical 
facj-ii^es toward which the participants did not show consensus as to 
satisfaction. Appendixes include related program material. . 
(Author) 
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FINAL REPORT 
AERA 1971 Research Training Sessions 



This is a final report of the AERA 1971 Research Training Sessions. 
This report includes the following sections: 1) background of training 
session activity, 2) overview of 1971 training session activity, 3) director's 
report and evaluation of each session, 4) the final fiscal report, and 
5) long term evaluation. 

The training sessions were scheduled in conjunction with the Annual 
Meeting of the American Educational Research Associi^tion in New York; Eight 
five-day intensive training sessions were staged prior to the February 4-7, 
1971 Annual Meeting.* Six of the training sessions were he3.d in the New 
Yorker Hotel in New York City, one was held at the Greyston House, Columbia 
University and one was held at the Educational Testing Service, Princeton, 
New Jersey. All of the sessions were scheduled concurrently from January 30- 
February 3, 1971 under the supervision of Dr. Joe L, Byers, Director of the 
Reseat ^tih Training Program, 1971. 

Objectives The training sessions are designed to improve the 
substantive knowledge and research skills of experienced educational researchers. 
Many of the research training programs funded by the Office of Education to 
date have been directed toward the training of new researchers. Few of these 
programs are suited to the needs of individuals who have completed their 
formal professional training and who, in turn, are training new researchers. 
Recent advances in methodological and substantive issues in educational 

*The original proposal requested partial support for ten sessions. ^6 of 
those session directors declined to hold the sessions when notified that the 
sessions would be partially self-sustaining. 



research necessitate that every educational researcher continue to receive 



research training. The AERA Research Training Sessions are designed to meet 

this need. The objectives of the AERA Research Training Program are best 

expressed in the following statement issued by the 1968 Presessions Committee 

and reaffirmed by the 1971 Program Committee: 

"Supervised training in the technical skills used by the 
educational researcher is generally available to him only 
at considerable expense and inconvenience once he has 
completed his gtaduate education and has assumed full 
professional responsibilities. Often he must leave his 
work for an extended period of time and travel to find 
those competent to instruct him. Much of the expense 
and inconverAience can be spared him if instruction in 
research skills is condensed into short training ses- 
sions held either before or after the Annual Meeting 
of AERA." 



"The purpose of the AERA Research Training Session 
Program is to train educational researchers in funda- 
mental research skills, e.g., experimental design, 
statistical analysis, survey techniques, measurement 
theory and techniques, electronic data processing, the 
functions of the computer In. research and research 
managements The Research Training Sessions are 
intended to be instructional or disseminative of es- 
tablished research techniqves as opposed to generative 
of new substantive problems or di7:ections for research 
in some particular area. The latter function is 
considered to be the purpose of symposia and conferences 
and hence, It falls within the scope of the Annual 
Meeting of AERA and the activities of other professional 
organizations." 



"Preference for participation in any Research Training 
Sessions will be given to reseiiirchers who hold a 
doctorate. This decision was made on the assumption 
that persons not holding a doctorate still have ample 
opportunities to improve their research skills while 
pursuing an advanced degree. On the other hand, the 
character of graduate education makes it relatively 
inaccessible to persons holding. a doctorate who have 
assumed full professional responsibilities." 



The 1971 AERA Research Training Sessions can trace their origins to informal 
meetings of one or two days duration involving a relative number of selected 
researchers prior to the 1964 and 1965 Annual Meetings. These sessions were 
not widely publicized and did not have the training of researchers as their 
primary mission. In 1966 the prototypical "presession" was held as one of a 
group of three meetings in the tradition of the previous , preconvencion meet- 
ings. The 1966 presession which set the precedent for future research 
training sessions, under the direction of Richard E. Schutz, dealt with the 
subject of experimental design. It was the first five-day presession 
sponsored by AERA and was, in addition, the first formal research training 
program completed under Title IV of the Elementary and Secondary Education 
Act of 1965. 

Because of the success and acceptance of the 1966 presession, 
coupled yith a growing interest of AERA members in the possibility of 
expanding and formalizing ether presession meetings, AERA sponsored a program 

of six courses in the 1967 presessions program. The 1967 presession program 
was supported in part by a grant from the U.S. Office of Education xinder 
Title IV of ESEA, 1965. Approximately 500 researchers applied for the 
program and somewhat more than 300 researchers actually participated. 

The highly positive response to the 1967 presession program led to 
greatly expanded programs of. eleven presessions in 1968; twelve sessions in 
1969; and ten in 1970. All were supported in part by grants from the U.S. 
Office of Education under Title IV of ESEA, 1965. The nxmiber of participants 
increased from 300 in 1966 to 550 in 1967 and 1968 and over 400 in 1970. 
Approximately seventy-five percent of the applicants were employed in college 
or university positions. ; Sixty-five percent of the applicants possessed 
doptorate degrees and thirty-two percent held the master's degree. Information 
on reisearch productivity revealed that the applicants had an average of 3.8 
published articles in scholarly journals, ahd^had directed ah average of one 
fimded research project. ^ 



During the five years in which AERA has conducted Research Training 
Sessions the response to these sessions has been consistently high. 
Applicant response reflects the need felt by educational researchers for 
the kind of re-training provided by the research train:^ns program. Due to 
the cutbacks in federal funding for general educational research, development 
and training programs, it is essential that programs such as the AERA research 
training sessions be continued. 

In the winter of 1970 AERA President Dr. R. M. Gagne appointed the 
Research Training Presession Committee. Its mejob^rs^were drawn from each of 
the divisions of the Association. The individuals selected were Dr. Donald 
Willower, Pennsylvania State. University ; Dr. John S. Mann, John Hopkins 
University; Dr. Thomas J. Shuell, State University of New York, Buffalo; 
Dr. Calvin Dyer, University of Michigan; Dr. Ray E. Hosford, University of 
California, Santa Barbara; Dr. Timothy Smith, John Hopkins University; 
Dr. Elizabeth Cohen, Stanford University;, and Dr. Joe L. Byers, Michigan 
State University, Chairman. 

The committee decided to follow the procedures of previous years and 
publish a "call" for presession training proposals in the Association's 
newsletter. Accordingly, a notice was placed in the Jferch 1970 issue of the 
Educational Researcher inviting interested individuals to submit proposals. 
In addition, various members of the committee whose research competence was 
outstanding urged colleagues to submit proposals. As a result of these 
efforts, the committee received twenty-eight proposals to evaluate. 

As proposals were received they were screened by the chairman and sent 
to a Divisional representative for detailed review. He. in turn obtained 



additional eyaltiations from trusted colleagues. Each proposal was rated by 
the following ten criteria: 

1. The general capability and experience of the director and his staff. 

2. The adequacy of the staff in terms of size and distribution of skills 
demanded by the objectived set forth in the training proposal. 

3. The importance or need for additional training on the topic. 

4. The appropriateness of the topic and goals to the presession training 
format . 

5. The clarity and specificity of the objectives. , 

6. The relevance of the objectives to general goals. 

7. The appropriateness of the objectives for the expected audience. 

8. The clarity and extent of instructional planning and its relevance 
to the objectives. 

9. The adequacy of^the evaluation procedures proposed. 

10. The general value of the topic to educational research. 

The divisional representative took the comments of his advisors along 
with his own judgments on each proposal and brought them with him to the meeting 
of the committee held in July 1970 in Chicago. 

When the committee met each proposal was presented by the divisional 
representative who had evaluated it. It was then discussed by the committee 
as a whole. Each proposal was then rated on its merits for tentative support 
or rejection. Following this first round in which proposals lacking substantive 
merit were eliminated a ranking of the remaining proposals was made. Factors 
contributing to this final ranking were 1) newness of the topic and 2) breadth 
of divisional and/or topic representation. The committee believed that ^ 
presessions which had been sponsored several times in the past should not be 
ranked as high as those Involving a new topic or methodology. Also the 
committee tried to provide a program of presession topics covering a broad 
spectrum of divisional Interests. With these criteria In mind, the committee 
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• selected the eight proposals for AERA sponsorship. 

The titles and iTiames of the directors of the eight sesslor^.v held this 
year are as follows: 

1. Systems Research for Counseling and Educational Environments 

T. Antoinette Ryan, University of Hawaii 

2. Educational Objectives; Formulation, Appraisal and Assessment 

W. James Popham and Eva L. Baker, UCLA 

3. The Psychology of Written Instruction 

Ernst Rothkopf and Lawrence Frase, Bell Telephone Labs. 

4. Nonparametric Methods and Related Post Hoc Procedures 

Maryellen McSweeney and Andrew C. Porter, Michigan State 

5. Operations Analysis Techniques in Educational Planning and 

Administration 

George S. Tracz, OISE and James E. Bruno y UCLA 

6. Individual Differences, Learning and Instruction 

Frank H. Farley, University of Wisconsin 

7. Multivariate Statistical Analysis in Educational Research 

Charles E. Woodson, University of California, Berkeley 

8. The Research Component of Black Studies 

LaMar P. Miller, New York University 

Evaluation Similar training sessions have been held, with USOE 
financial assistance, for five years. This year's training sessions were 
the Association's first ^attempt at self-sustaining research training programs. 
A $50.00 registration fee was assessed of participants in the 1971 training 
Sessions, and a proposal for partial funding to cover the if^ttmaining costs 
was submitted to both the USOE and the NSF* Despite the $50.00 registration 
fee the attendance rate for this year's sessions was neither more or less than 
previous years. The average drop-out rate (cancellations included) for the 
1^'71 sessions was 11%. 365 of the 413 accepted apr^^icants actually attended 
the sessions.* (Comparative statistics on the drop-out rate of previous years 
are not available) • Attached is a chart of the sources of the registration 
fee. Three categories of payment sources are indicated » Because not all 
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participants had paid by the date of this analysis, the total number of payments 
does not equal the total number of participants. 

In addition to the Immediate post session evaluation conducted by 
each presesslon director and described In his report, the Association conducted 
a long term follow-up evaluation of the participants. This evaluatl.on, 
conducted by mall, attempted to determine the long term (nisie-iDonth) conse- 
quences of the presesslon experience on the scholarly activities of its 
participants. This evaluation is included in the report. 
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SYSTEMS RESEARCH FOR COUNSELING AND EDUCATIONAL ENVIRONMENTS^ 



T. A. Ryan . , 
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The research reported herein was performed pursuant to a aubcontract with 
the American Educational Research Association. Points of; view or opiniona 
stated do not^ therefore, necessarily represent official Atoerlcan Educational 
Research Asaoc let ion position or policy. 
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SUMMARY 



This is a report of a five-day research training session; held in 
New York City, New York from January 30 to February 3, 1971 under sponsorship 
of the American Educational Research Association with support from U. S. 
Office of Education. The purpose of the training session was to develop 
and improve research competencies of individuals engaged in counseling, 
counselor education, and related research. The program purpose was 
implemented in two primary aims: (1) developing participants* understanding 
of systems research principles and concepts, and (2) helping participants 
acquire proficiency in systems research skills and techniques. 

■R/enty-two trainees were selected to participate in the training 
session. Program activities included didactic instruction, individualized 
and group 'activities, and supervised practice. 

Immediate evaluation using Internal criterion measures revealed 
attainment of program objective at or near performance level. Follow -up 
will be conducted to assess Icng-range results. Program evaluation indi- 
cated satisfaction with all aspects of organization and administration of 
the training session, with the exception of allocation of time and physical 
facilities towards which the participants did not show conaenaus as to 
satisfaction. 
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I. Introduction 

A. Problem 

This is a report of a five-day research training Session conducted 
in New York City, New York from January 30 to February 3, 1971. The session 
was one of eight research training programs sponsored In 1971 by American 
Educational Research Association with support from U. S. Office of Education 
for the purpose of achieving improvement in education through research. 
The prescasions were designed to meet the needs of individuals whose full- 
time professional responsibilities preclude possibility of long-t6rm 
training in specialized and advanced research techniques. The presesslon 
in systems research was designed to meet research needs of counseling 
persotTnel, counselor educators, educational psychologists and researchers 
in related areas. 

B. Statement of Need 

The need for improvement and innovation in counseling, counselor 
education and related areas has intensified as social, political and 
economic factors have created new problems and greater challenges for the 
educational community (McDaniels, 1967; Riccib and Walz, 1967; Stoughton, 
1965; Wolfbein, 1967). With the adoption of standards for counselors and 
counselor educators the need for research skills was intensified (American 
School Counselor Association, 1965; Association for Counselor Education 
and Supervision, 1965). The standards carry an implicit mandate to the 
profession to make a searching analysis of goals and study of the ways in 
which to achieve goals most effectiveily. The research training session 
was designed to equip selected personnel with the research skills needed 
to implement needed innovations and improvement in the field. 

C. Rationale 

Counseling, counselor education, and related educational programs 
can be conceptualized as systems. Therefore;, it can be assumed that 
improvement in couuseling, counselor education, and related educational 
programs can be achieved through application of techniques of systems 
research to these areas of educational endeavors. 

In counseling, counselor education, and related areas these are 
needs to investigate problems and to arrive at best possible solutions to 
these problems. The systems research techniques of analysis, synthesis, 
modeling, and simulation can be eixq>loyed to meet these needs. 

The acquisition of proficiency In using systems techniques can 
be accomplished in a short-term training session. 

D. Purpose and Objectives 

The purpose of the presesslon was to improve counseling, counselor 
education, supervision, and related areas through research. The program is 
designed to train selected participants Id use of systems approach for 
planning and evaluating counseling, counselor education, supervision, snd 
related areas, 

15 
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IWo primary alms implemented the program purpose: . 

(1) development of participants* knowledge and understanding of 
systems research concepts and principles. 

(2) development of participants* proficiency in using systems 
techniques for planning and evaluating counseling and counselor education. 

Objectives implementing the presession were: 

(1) given a multiple choice objective teat, participants would 
demonstrate understanding of systems research concepts by being able to 
select from alternative endings to one ending which best coa^)Ieted the 
statement of definition or illustration of basic systems concepts such as 
system^ analysis^ synthesis^ simulation^ model ^ an^isynthesis^ flowchart^ 
synergism^ logistics^ and fidelity^ with an acceptable performance level set 
at 80 percent correct responses in a given time period. 

(2) given a multiple choice objective test^ participants would 
demonstrate understanding of principles of systemfii research by selecting 
from among alternatives the one ending which best completed statements of 
principles or illustrates principles such as the rules for coding^ lettering^ 
and signal paths, with acceptable performance level set at 80 percent correct 
responses in a given time period. 

(7) given a narrative description of a problem situation the 
participant would be able to convert this word description into a flowchart 
model with correct element identlf icationj use of symbols, descriptors, 
signal paths, blocks, coding and lettering, with acceptable performance 
level in a given time limit set at 80 percent agreement with problem solution. 

(4) Given a flowchart model of a problem situation, the 
participant would be able to convert this model into a narrative description, 
with acceptable performance level in a given tiine period set at 80 percent 
agreement with problem solution. 

(5) given criteria for defining behavioral objectives, and a 
set of objectives, participants would be able to determine which objectives 
were stated in behavioral terms and the extent to which criteria for defining 
objectives behaviorally were satisfied. 
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II. Method 

A. Design 

The training program was designed to provide an integrated 
sequence of learning experiences. It was assumed that achievement of 
program objectives would be relat.3d to (1) participant selection; (2) infor- 
mation presented; and (3) practice provided. 

Selection of participants was made to obtain a highly homogeneous 
group. Amount of kind of information presented to participants were 
controlled through iise of assigned readings^ mcdla^ staff presentations^ 
and instructional materials. The acoount and kind of practice were controlled 
through the schedule and sequence of practice sessions^ use of graduated 
practice exercises^ and staff supervision of individualized and group 
problem-solving activities. 

The research training program was five days in lengthy from 
January 30 to February 3^ 1971. Dally sessions were held from 8:00 A. M. 
to 5:00 P.M. Staff were available to provide individualized Instruction or 
consultative services in the evenings. 

B* Participants 

There were numerous applicants for the presesslon on systems 
research in counseling^ counselor education and related areas. The maximum 
number of enrollees which could be accepted was twenty-two^ since training 
packets were utilized in th« program and only twenty-two packets were 
available. 

Applications were evaluated on the basis of the following criteria: 

(1) Capability for implementing systems research techniques in 
counseling^ counselor education^ or related areas. 

(2) Potential for developing understanding of systems concepts^ 
and proficiency in using systems techniques. 

(3) Interest in acquiring Information about and skills of systems 

research. 

Method of Selection 

Notice of the eight presesslons sponsored by the American 
Educational Research Association was printed in the December^ 1970 issue 
of Educational Researcher^ the official AERA Newsletter. Letters of 
invitation to participate in the presession on sy steams research in counseling^ 
counselor education and related areas were mailed by the presesslon director 
to a selected list of potential candidates whose ba<:kground of education 
and experience satisfied criteria for research training. Respondents 
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accepting the invitation to participate were pre-enrolled. Applications 
submitted in response to the public announcement of the presession were 
evaluated individually as received on the basis of selection criteria. In 
selecting participants^ there was no discrimin^^tion on account of sex, race, 
color, or national origin. Participant roster and ongoing projects are 
presented in Appendix A. 

Each applicant was notified of the action talcen on his application. 
Participants selected for the training session were required to file an 
Enrollment Form. 



Participant Characteristics 

Participants came from thirteen states and Canada, included 19 
males and 3 females and represented higher education, local school districts, 
private schools, and government agencies. Out of twenty-two participants, 
thirteen held the doctoral degree. Distribution of participants by sex, 
highest educational degree, place and nature of employment is given in 
Appendix B. 

€• Staff 

It was assumed that staff competencies and qualifications should 
be sufficient to allow foi effectively (1) presenting information on systems 
resesarch concepts and principles; (2) showing the relevance of systems 
research to counseling and counselor education, and related problem areas; 
and (3) providing supervision and consultative services to individuals 
and groups engaged in learning activities on systems research. 

Presession staff included the director, who implemented project 
management and instructional functions, and five instructors (See Appendix C). 

D. Training Program 

The training program designed to achieve presession objectives 
was five days in length, with daily sessions from 3:00 A.M. to 12:00 NOON, 
and 1:00 P.M. to 5:00 P.M. 

Prior to the start of the presession, training was Initiated. 
Upon receipt of his Enrollment Form, a materials packet was sent to each 
participant, including seven references for required pre^conference reading, 
syllabus, reference list and staff directory. The syllabus is given in 
Appendix D, and the reference list in Appendix E. 

The presession opened with an orientation to the program. This 
was followed by a pretest to determine extent to which terminal objectives 
might already have been achieved by participants. After completion of the 
pretest the program included seven major elements (1) presentation of 
concepts and principles to reinforce pre-conference reading and clarify 
misunderstandings; (2) initial instruction in skill development; (3) advanced 
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Instruction in skill developmcat; (4) practice in application of systems 

techniques; (5) post assessment to determine extent to which objectives 

had been achieved and provide bases for necessary individualized Instruction; 

(6) application of systems techniques to solve a real-life problem; and 

(7) presentation of models implementing principles and techniques of systems 
research. The daily schedule is shown in Appendix F* 

Activities designed to achieve Aim 1, development of understanding 
of systems concepts and principles included assigned reading^ lectures^ 
slide-tape^ and f ilmstrip-tape presentations^ films, individualized activities 
using programmed materials, and supervised practice on workbook exercises. 

Activities designed to achieve Aim 2, acquisition of proficiency 
in applying systems research techniques included supervised practice on 
individual and group problems, and supervised practice with advanced exercises. 
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Ill, Results 

A. Evaluation of Participant Performance ^ 

Two measures were taken to evaluate participant performance 
against program aims: an objective pre- posttest^ and self evaluation by 
participants. 

Evaluation of the presession in terms of participant achievement 
of training objectives was accomplished by comparing prf;-> and post- 
instructional performance on a test intended to sample behaviors implementing 
program aioa. The same instrument^ which was administered for pre* and 
post -instruction testing^ contained three subjects^ all of which were 
intended to sample behaviors relating to Aim 1^ understanding of concepts 
and principles of systems research. Subtest 1 was concerned with basic 
concepts and principles of systems theory and research. Subtest 2 was 
concerned with developing, a flowchart model. Subtest 3 was concerned with 
interpreting another flowchart model. 

Table 1 (Appendix G) shows the group profile for mean scores on 
the pre- and posttest by objective component. The mean scores on all three 
subjects for the posttest administration showed positive gains as compared 
to the pre-test mean scores, as indicated by the increase from 15-94 to 
17-75 on Subtest 1, 6-67 to 12-06 on Subtest 2, and 13-81 to 15-31 on 
Subtest 3. 

Acceptable performance criteria were defined for Aim 1^ developing 
understanding of systems concepts and principles, this criteria relating to 
the objectives of the pvesession where 80 percent agreement with the correct 
answer in each subset constituted the desired end result. The degree to 
which the criteria is satisfied is presented in Appendix G, Table 2. 
Inspection or Table 2 reveals that, for the sroiip as a whole, the performance 
criteria were met for all three subtests, where the percentages of agreement 
with the correct answers were 89 percent, 80 percent, and 90 .percent 
respectively. These are significant gains compared to the level of agreement 
with the correct answers during the pretest in which the percentages were 
80 percent, 45 percent, and 82 percent respectively. 

No Immediate objective test was taken to sample behaviors relating 
to Aim 7:, participant proficiency in applying systems techniques. A follow- 
up is pleaaed to evaluate the program against this objective. 

Self-evaluations against Aims 1 and 2 were taken by eliciting from 
participants responses to indicate how participants felt about progress 
they made toward training pbjectivea. Table 3, Appendix G, reports percent 
of participants who felt a significant gain in knowledge or increase in 
skill proficiency resulted from participation in the training program. 
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inspection of Table 3 indicates 94 percent of participants felt 
the program resulted in their acquisition of knowledge about systems research, 
and 88 percent felt the program increased proficiency in using systems 
techniques. 

B. Evaluation of Program Organization and Administration 

A program evaluation was made to determine extent to which program 
components contributed to effectiveness of the presession. Data were 
gathered to evaluate learning activities, instructional materials, program 
content, and program organization. 

Participants rated program learning activities on a four-point 
scale to indicate degree to which the activity contributed to achievement 
of program effectiveness. Mean ratings are reported in Appendix TabXe 4* 
Examination of the data reported in Table 4 reveals that all activities 
were rated above the mean expected by chance. The activity deemed most 
worthwhile was the individual problems, followed closely by individual 
conferences with staff. Audio-visual presentations were rated lowest. 

Evaluation of instructional materials was made by participant 
rating on a four«point scale of six references which were required for the 
program. Mean ratings are reported in Appendix Table 5. Examination 
of Table 5 reveals that all references were rated above the chance mean. 
The reference rated as most worthwhile was Systems techniques for programs 
of counseling and counselor education by T. A. Ryan, with the next highest 
rating for Systems design in the development of counseling and guidance 
programs by R. E. Hosford and T. A« Ryan. 

Program content was evaluated by participant rating on a four* 
point scale of each program unit in terms of contribution to program aims. 
Mean ratings are reported in Appendix Table 6. Inspection of Table 6 
reveals that units considered most valuable were Problem g From Real Life 
Environment, Illustrations of systems research in counseling, testing, 
school negotiations, counselor education t and Model for producing a systems 
model. The unit rated lowest was the Nonsense problems. All units were 
rated above the chance mean. 

Program management was evaluated by participant ratings of aspects 
of program organization and management relating to information, meals 
and lodging, staff qualifications, time utlizatlon, and climate for learning. 
Participant ratings of program management are reported in Appendix H, 
Table 7. Results in Table 7 indicate general satisfaction with program 
information, living accomodations, staff competencies, and learning climate. 
There was no consenaus regarding facilities and time allocation and utlization, 
although more regarded the facilities or time factor as satisfactory than 
unsat is f ac t cry . 
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IV. Discussion 

A. Purpose 

The primary pitrpose of the American Educational Research Association 
prescssion on systems research for counseling^ counselor education^ and 
related areas was to develop and improve research competencies of counseling 
specialists^ counselor educators^ supervisors^ educational psychologists, 
and researchers engaged in research in counseling, counselor education and 
related areas. The program purposes were implemented in two primary aims: 
(1) development of participants' knowledge and understanding of systems 
research concepts and principles; and (2) development of participants* 
proficiency in using systems techniques for planning and evaluating counseling 
and counselor education. 

B. Results 

Analysis of results from criterion tests indicates that the first 
aim, developing participants' knowledge and understanding of systems principles 
and techniques was achieved. 

Comparison of pre- and posttest scores indicates that participant 
understanding of concepts and principles increased significantly as a 
function of the presession training on all three subtests. When performance 
on the criterion tests is compared against a standard of acceptable performance 
(Appendix G, Table 2), it is noted that the standard was met for all three 
subtests for the group as a whole, whereas the level of agreement with the 
correct answers was only marginal before the presession. No test was 
administered to assess understanding of behavioral objectives. This is 
based on the rationale that the nature of the program assusaed an initial 
understanding of these concepts (1*) a&d therefore included only a minimum 
of learning activities aimed at developing or increasing understanding of 
behavioral. 

No direct measure was taken to assess participant progress toward 
attainment of Airi 2, developing proficiency in using skills and techniques 
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- The program of instruction iu the presession assumes a prior 
understanding of certain basic concepts and principles, and ability to 
perform certain activities with ease and competence. In order to derive 
maximum benefit from the training program, participants must have a 
thorough understanding of the language of systems research, and must be able 
to operationalize mission goals and^to define behavioral objectives. It 
is assumed that before the presessid^ begins participants will be capable 
or defining problems, stating objectives in behavioral terms, and 
identifying alternatives to inq;>lement the objectives. The references, listed 
are intended to provide a means by which participants can acquire the 
prerequisite knowledge and skills which are assumed for this program. 
Reference annotations are provided to assist in directing reading activities 
so that optimum use can be made of participants' reading time pribr to the 
start of the presession^ Quoted from page 1, Selected Refe rences, AERA-024. 
^ February 16, 1970. ' ' 
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of systems research. Evaluating participants against this objective 
calls for performance testing^ which was not attempted In the training 
session. A follow-up will be conducted to attempt to get an Index of 
participant proficiency In terms of quality and quantity of skill 
Implementation In the real life environment. An indirect measure was 
taken to give an immediate indication of the extent to which this aim 
may have been achieved. This was done through participant self -evaluation 
against the criterion and staff rating. The self evaluations by participants 
rei^ealed that 88 percent felt they had acquired a significant amount of 
proficiency in using systeins skills and techniques. This impression was 
confirmed by staff observation. 

The evaluation of program management suggests that all learning 
activities were meaningful and contributed to achieving program goals. 
The participant rating of reading materials suggests that the selection 
of references for the course was viable as well as worthwhile. All ratings 
were above the chance mean. Participants liked working on Individual 
Problems best and found as most valuable reading Systems techniques for 
programs of counseling and counselor education by T. A, Ryan. In terms of 
program unit, the Problem: .From Real Llfa Environment was most preferred. 

Responses to the program tianagement evaluation generally reflect 
a desire for more tlme^ especially in respect to individualised activities 
and to meeting with staffs This suggests the posslbilUy of offering a 
seven-day training session^ which might be given at a time other than 
immediately preceding the annual meeting. It is suggested that the desire 
on the part of . participants for more time reflects a desire for more training 
rather than management weakness. 

There was some dissatisfaction with the physical facilities. This 
reflects the reaction to unscheduled moves^ lack of facilities for jeff active 
use of audio-visual equipment^ poor ventilation^ Inadequate lighting^ 
overcrowding^ high noise interference^ and related inconvenience and 
obstacles to learning. Aside from the negative physical facllltes^ the 
climate for learning appeared to be conducive to reaching program goals. 

The participant comments to an outside evaluation (Appendix I) 
suggest that insufficient time was spent on individualized instruction and 
problems and perhaps too much time on group activities and certain basic 
concepts. Preference was also stated for more feedback from the staff. The 
staff assistance problem was largely a function of bad physical setup^ and 
could be overcome with ease given more adequate facllitiea. Many participant 
also deemed the presesslon stimulating and worthwhile. 

In general the evidence suggests that the program was viable 
and attainment of the long range goals^ Improvement of education through 
research^ should be realized. 
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AJ^FiNDIX A 



UNIVERSITY OF HAWAII 



EDUCATION RESEARCH AND DEVELOPMENT CENTER 



Honolulu, Havall 96822 



1776 University Avenue 



AMERICAN EDUCATIONAL RESEARCH ASSOCIATKHJ 1971 IRESESSION 
SYSTEMS RESEARCH FOR COUNSELING AND EDUCATIONAL ENVIRONMENTS 

January 30 - F^bpruary 3, 1971 
New Yorker H^^l, Nev York 



PARTICIPANT ROSTER 



1 . Dr • David A. Anderson 
Project CoordinatOA. 
Joliet Junior College 
Kt. iE^3, Houbolt Avenue 
Joliet, Illinois 60436 

2. Dr. Leon H. Belcher 

Director of Institutional Research 
Texas Southern University 
Houston, Texas 7700A 

3. Dr. James W. Bonsnarito 
Associate Professor of Special 

Education 
Illinois State University 
Normal, Illinois 61761 

4. Mr, Irving L. Broudy 
Program Director 
Educational Testing Service 
Rosedale Road 

Princeton, Nev Jersey 08540 

5. Dr. George H. Charlesvorth 
Director • Guidance Research and 

Statistics 
Stratford School System 
5344 Main Street 
Stratford, Connecticut 06497 

6. Sister El:lzabeth Ann Glysh 
Director of Special Services 
Alverno College 

3401 S. 39th Street 
Milwaukee^ Wisconsin 53215 

7. Dr. Irvine C. Gordon 
Vrofessor of Education 

I rairie View A & M College 
irrairie View, Texas 77445 



8. -Mr. Dvnstan L. Haettenschwiller 

Counselor 

SEEK - State University College 

at Buffalo 
1300 Elmwood Avenue 
Buffalo, New York 14222 

9. Dr. Robert C. Harris 
Head, Northwestern Centre 

The Ontario Institute for Studies 

in Education 
10 S. Algoma Street 
Thunder Bay, Ontario, Canada 

10. Dr. Halas L. Jackim 
Professor of Education 
State University of New York 
Oswego, New York 13126 

11. Dr. Robert Morgan 

Director of Counseling Services 
Bedford School System 
Bedford, Massachusetts 07130 

12 • Dr. P. Kenneth Morse 

Associate Professor of Dental Education 
Medical College of Georgia 
Augusta, Georgia 30904 

13. Dr. Dennis E. Nelson 
Assistant Professor of Education 
frarie View A & M College 

5303 Lookout Mountain Drive 
Houston, Texas 77040 

14. Dr. Ralph E. Packard 
Director, Counseling Center 
University of Utah 

Salt Lake City, Utah 84112 
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APPENDIX B 



PARTICIPANT CHARACTERISTICS 



Sex 

Male 
Feaiale 



Number . 

19 
3 



Education 

Ph.D. 
Other 



Number 

13 
9 



Place of Residence 



Region 
East 



State ^ 



New York 
New Jersey 
Massachusetts 
Connecticut 



Southwest 



Texas 



Southeast 



Midwest 



West 



Canada 



Georgia 
Florida 



Illinois 

Indiana 

Wisconsin 

Ohio 

Kansas 



Utah 



Ontario - 



Number 



6 
1 
1 
1 



1 * 
1 



2 
1 
1 
1 
1 



Naturel of Employment 
Employer Posi tion Number 



Higher. Education 

Professor 
Assoc. Professor 
Asst. Professor 
Director 
Coordinator 
Chairman . 
Counselor 

Public School 

Director 

Coordinator 

Associate 



3 
2 
3 
5 
I 
1 
I 



3 
1 
2 
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APPBNDIX C 

tlNIVBRSITY OP HAWAII Honolttlu, Hwall 96822 

EDUCATION RESEARCH & DEVELOPMENT CENTER 1776 University Avenue 

AM8BXCAN EDUCATIONAL RE8EARCB ASSOCIATION 1971 PRESESSION 
SYSTEMS RESEARCH FOR COUNSEUMG AND EDUCATIONAL ENVIRONMENTS 

January 30 • February ^ 1971 
Mew Yorker Hotels New York 

STAFF DIRECTORY 



Director 

T* A» Ryan^ Researcher/Professori Education Research and Development Center, 
University of Hawaii, Honolulo, Hawaii 96822 



Staff ' i- , 

Donald Hays, Director, Pupil Personnel Services, Fullerton Union High School 
District, District Education Center, 211 West Connaonwealth Avenue, 
Fullerton, California 92632 

Ray E« Hosford, Associate Professor of Education, University of California, 
Santa Barbara, California 93106 

Leonard C» Silvern, PrestdentV Education and Training Consultants Company, 
12121 Hllshire Boulevard, Los Angeles, California 90025 

Nonnan R« Stewart, Professor, College of Education, Department of Counseling, 
Personnel Services and Educational Psychology, 448 Brickson Hall, 
Michigan State University, East Lansing, Michigan 48823 

Bob B. Wlnborn, Professor, College of Education, Departmsnt of Counseling, 
Personnel Services and Educational Psychology, 436 Erickson Hall, 
Michigan State University, East Lansing, Michigan 48823 
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APPENDIX D 



UNIVE R SITY OF HAWAII H onolulu, Hawaii 96822 

EDUCATION RESEARCH. AND DEVELOPMENT CENTER 1776 University Avenue 

AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 1971 PRESESSION 
SYSTEMS RESEARCH FOR COUNSELING AND EDUCATIONAL ENVIRONMENTS 

January 30 - February 3, 1971 
New Yorker Hotel , New York 

SYLLABUS 

I, Nature of ^esession 
A, Description 

1, This presession in systems research is one of eight research training 
sessions of feried In 1971 by American Educational Research Association* 
The 1971 Presesslons are self-supporting. There is a $50 enrollment 
fee for each participant, 

2, This training session is designed as an advanced program focusing on 
the use of systems research for planning and evaluating counseling, 
counselor education, supervision and related programs, 

3, The program has been planned to equip counseling specialists, counselor 
educators or supervisors, educational psychologists , and researchers 
performing substantive research in counseling, counselor education, 
supervision or related areas with practical skills and theoretical 
knowledge essential for implementing systems research at local 
district, state department or university levels, 

4, The course of study deals with conceptualization of systems research, 
application of systems research, techniques of systems research, and 
practical uses of systems research, 

B* Staff 

1, Director: T, A. Ryan, University of Hawaii 

2, Instructors: Donald G* Hays, Union High School District, California 

Ray E, Hosford, University of California, Santa Barbara 
Leonard C, Silvern, Education and Training Consultants, 

Company, Los Angeles 
Norman R. . Stewart, Michigan State University 
Bob^B, Witiborn,.-Michig.an State University 

C, Participants > 

The session will be open to individuals in public schools, state departments 
of education, and colleges and universities who satisfy the following 
criteria: 

1, Employra^t as counseling specialists , counselor educators or supervisors 
educational psychologists, or researchers with responsibilities for 
performing substantive research in counseling^, counselor education, 
supervision or related areas. AERA-019 
O . 12/18/70 
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2. Education and experience background to give evidence of ability 
, to profit from instructional program. 



Purpose and Objectives 

L. ABRA rationale for conducting presessions is based on assump- 
tions that 

a, significant benefits accrue from improved and expanded 
educational research and 

b. short, intensive in-service training is effective for equip- 
ping those in professional roles H»ith advanced knowledge 
and specialized research skills. 

2. Purpose of the presession on systems research is to improve 
counseling, counselor education, supervision, and related areas 
through research. This program is designed to train selected 
participants in use of systiems approach for planning and evalu- 
ating counseling, counselor educationj supervision, and related 
areas, 

3. Primary aims of the presession are: 

a^ to develop participants' knowledge and understanding of 
systems research concepts and principles. 

b. to develop participants' proficiency in using systems tech- 
niques for planning and evaluating counseling and counselor 
education. 

4. Objectives implementing the presession aims are: 

a. Given a multiple choice objective test, participants will 
demonstrate understanding of systems research concepts by 
being able to select from alternative endings the one 
ending which best completes the iatatement of definition 
or illustration of basic systems concepts such as system, 
analysis, synthesis, simulation, model, anasynthesis, flow- 
chart, synergism, logistics, and fidelity, with an accept- 
able performance level set at 80% correct responses in a 
given time period,. 

b. Given a multiple choice objective-test participants will 
demonstrate understanding of principles of systems research 
by selecting from among alternatives the one ending which 
best completes statements of principles or illustrates 
principles such as the rules for coding, lettering, and 
signal paths, with acceptable performance level set at 

807. correct responses in a given time period. 

c. Given a narrative description of a problem situation, the 
participant will be able to convert this word description 
into a flowchart model with correct element identification. 
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use of symbols, descriptors, signal paths, blocks, coding 
and lettering, with acceptable performance level in a given 
time limit set at 80% agreement with problem solution. 

d. Given a flowchart model of a problem situation, the parti- 
cipant will be able to convert this model into a narrative 
description, with acceptable performance level In a given 
time period set at 807o agreement with problem solution. 

e. Given criteria for defining behavioral objectives, and a 
set of objectives, participants will be able to determine 
which objectives are stated in behavioral terms and the 
extent to which criteria for defining objectives behaviorally 
are satisfied. 

II. Program Outline and Ac tlvi tea 

A. Outline 

1. Systems research 

a. concepts and principles 

b. definitions 

c. background 

2. Techniques of systems research 

a. analysis 

b. synthesis 

c. modeling 

d. simulation 

3. Practice in using systems research 

a. general problems 

b. counselor education and counseling problems, simple and 
complex 

4. Application of systems research to real«:life problem 

a. situations identified by participants 

b. systems techniques applied to real life situations 

B. Activities 

1. Program will be intensive and demanding, involving five full 
workdays In addition to Independent study and informal group 
sessions during evening hours . 

2. Activities will Include lecture, discussion, demonstration, and 
task groups. 

3. Supervised practice in use oiE systems research will occupy major 
part of program, with participants working individually an4 In 
groups on prepared problems. ^ 
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III. Requirements 

A. Participation 

!• Participants will be required to attend and to participate in 
daily lecture-'discussion periods. 

2. Participants will be required to participate in task groups. 

B. Reading 

1. Reading requirements will be determined accbrding to needs 
of individual participants. This training program assumes a 
starting background of prior knowledge and skill proficiency 
on the part of participants. The reading list has been pre- 
pared with this in mind and is intended' to serve the purpose 
of directing participants to sources, of information for use 
in overcoming specific knowledge or skill deficiencies. 

2. It is recommended that participants study the references in 
the Materials Packet. An individualized program of in-depth 
study should be undertaken by each participant according to 
individual needs for background knowledge and skill develop- 
ment, so all participants will be starting the program with 
prerequisite knowledge and skill capabilities needed to benefit 
from training. 

IV. Evaluation 

A. Participant evaluation will be based on 

1. pre- and posttest of use of basic-principles of systems 
research covered in the training sessions; and 

2. self -evaluation by participants. 

B. Program evaluation will be made through participant opinions con- 
cerning materials^ staff and organir'^ation. 
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APPENDIX E 

UNIVERSITY OP HAWAII Honolulu^ Hawaii 96822 

EDUCATION RESEARCH AND DEVELOPMENT CENTER 1776 University Avenue 

AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 1971 PRESESSION 
SYSTEMS RESEARCH FOR COUNSELING AND EDUCATIONAL ENVIRONMENTS 

January 30 • February 3, 1971 
New Yorkeir Hotels New York 

SELECTED REFERENCES* 



The prograui of instruction in the preseasion assumes a prior understanding of 
cei^ta in basic concepts and principles, and ability to perform certain activities 
with ease and competence • In order to derive maximum benefit from the training 
program, participant a must have a thorom^h understanding of t he language of 
systems research, and must be able to opera tlonalize mission goals and to define 
behavioral objectives . It is assumed that before the presesslon begins partici- 
pants will. be capable to defining problems, stating objectives in behavioral 
terms, and identifying alternatives to Implement the objectives. The references 
listed are intended to provide a means by which participants can acquire the 
prerequisite knowledge and skills which are assumed for this program. Reference 
annotations are provided to assist in directing reading activities so that optimum 
use can be made of participants' reading time prior to the start of the presession. 

^nathy, B. Instructional systems . Palo Alto, California: Fearon, 1968. 
A good overview of systems approach. Easy reading. Should be studied 
. by everyone to insure thorough understanding of the nature of systems research . 
T he appendix is particularly good. 

Bogus law, R. the new Utopians: A study of system dfslgn and social change . 
Englewood Cliffs, N.J. : Prentice-Hall, 1965 
An overview of systems approach. Intermediate reading level. 

Buckley, (Ed.) Modern systems research for behavioral scientists . Chicago: 
Aldine Publishing Co«, 1968. 

A collection of articles dealing with systems theory. Advanced reading. 

Carter, L, F. Systems approach to education: Mystique or reality. Educational 
Technology . 1969, 9, 22-31. 

Gives an overview of the systems approach, with discussion of pros and 
cons from using the systems techniques. 

Churchman, C. W. The systems approach > New York: Oelacorte Press, 1968. 

^his brief discussion of systems approach gives an' excellent overview of 
the tof:al systems concept , which involves . problem identification, objectives 
definition, alternatives. Identification and evaluation. This reference helps 
to point up the way in which flowchart modeling and simulation techniques 
implements the systems concept. Should be studied by all participants. 



^Items marked with asterisks are included in participAnts Materials Packet. 
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Cooley, W. W. and Hunrael, R. C. Systems approaches in guidance. Review of 
Educational Research , 1969, 39, 351-362. 
Relates systems techniques to guidance. Easy reading. 

Education^T^fechnol^fcy , 1969, 9, No. 3, 1-77. 

This special iasui^^f Educational Technology is devoted to counseling 
technology. 

Eraut, M. R. An instructional systems approach to course development. 
AV Communication Review , 1967, 15, 92-101. 

Relates the techniques of systems research to course development. 

Gagne, R. M. Educational objectives and human performance. In Krumboltz, J. D. 
(Ed.) Learning and the educational process . Chicago: Rand McNally, 1965. 
t'p. 1-24. 

Discusses definition of objectives. Easy reading. 

*Hosford, R. E., and Ryan, T. A. Systems design in the development of counseling 
and guidance programs. Pfersounel and Guidance Journal , 1970, 49(3), 
pp. 221-230. 

*Mager, R. F. Preparing instructional ob1ectives « Palo Alto, California: Fearon, 
1962. 

This book tells how to prepare behavioral objectives. It is mandatory that 
each participant be able to define objectives in behavioral terms. The 
principles discussed in thiis reference must be thoroughly understood by each 
participant. Each participant must be able to demonstrate proficiency in 
defining objectives behavioral ly. This can be accomplished by concentrated 
study of this reference, and practice in preparing behavioral objectives. 
Should be studied carefully by all participants. 

* Ryan, T. A* Systems techniques for programs of counseling and counselor 
education. In Silvern, L. C. (Ed.), Applying systems engineering 
techniques to education and training. Educational Technology . 1969, 9, 
1-17. _X 7 

This article describes the appli-CStion of systems techniques in counseling 
and counselor education. It provides ^a frame of reference for the pre- 
session. Easy reading. Excellent^ibliography on systems research. Should 
be read by all participants. The other articles in this issue of Educa- 
tional Technology are relevant to the presession topic. Casual reading is 
recoomiended. 

*Silvern, L. C. Systems engineering of education I; The evo lution of systems 
thinking in education ^ Los Angeles; Education and Training Consultants, 
1968. 

This is the basic text for the course . Pages 111-129 should be studied 
carefully by all participants. The program ssumes that participants will 
have read this material ^nd have a thorough understanding of the concepts 
presented in these pages. 
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^Silvern, L. C. LOGOS: A system language for flowchart iwvdeling. In Silvern, 
L. C# (Ed.), Applying systems engineering techniqxies to education and 
training. Educational Teeh n olo^y ^ 1969, 9, 18-23* 

Contains basic vocabulary for flowchart modeling. Should be studied by 
all participants. 

Thoresen, C. E. The systems approach and co^mselor education: Basic features 
and implications. Counselor Education and Supervision 1969. 
Discusses the application of systems techniques to counselor education. 

von Bertalanffy» L. Mo dern systems theory . New York: George Braziller, Inc., 
1968. 

Deals with systems theory. Advanced reading. 

Wierner, N. Cybernetics . Cambridge, Mass.: Massachusetts Institute of Technology 
Press, 1961. 

Deals with cybernetics aspect of systems research. Advanced reading. 



Wierner, N. Human use of human beings . New York: Doubleday, 1954. 

Deals with cybernetics in relation to systems research. Advanced reading. 
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EDUCATION RESEARCH AND DEVELOPMENT CENTER 
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1776 University Avenue 



AMERICAN EDUCATIONAL RESEARCH ASSOCIATinN 1971 PRESESSION 
SYSTEMS RESEARCH FOR COUNSELING AND EDUCATIONAL liNVIRCKMNTS 

January 30 - February 3, 1971 
New Yorker Hotel, New York 

COURSE OUTLINE 
(Saturday, January 30, 1971) 



Morning 

8:00-8: 10 

8:10-8:50 

8:50-9:05 

9:05-9:25 

9:25-10:00 

10:00-10:20 
10:20-11:00 

11:00-11:05 
11:05-11:15 
11:15-11:20 
11:20-12:00 
Afternoon 

1:00-1:40 

1:40-1:45 
1:45-2:25 
2:25-2:30 
2:30-3:00 
3:00-3:20 
3:20-4:30 

4:30-4:50 

4:50-5:00 
Evening 
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Opening T. A. Ryan 

Pre-Assessment T. A. Ryan 
Introductions 

Program Overview: Purposes and Procedures T. A. Ryan 



The Systems Approach: Concepts and 
Principles 

Recess 

Use o£ Systems Approach in Counseling 
and Educational Environments 

Question and Answer Period 
Model for Producing a System 
Question and Answer Period 
Defining System Goals and Objectives 



Behavioral Objectives for Counseling 
Environments 

Question and Answer Period 
Analysis as a Process 
Question and Answer Period 
Model for Producing a Model 
Recess 

LOGOS Language for Flowchart 

Modeling - individualized activity 

Question and Answer Period 

Closing 

Assignment: Bxeiccises 1^2^39^ 



T. A. Ryan 

R. £. Hosford 

Slide Tape 
D. G. Hays 

B. B. Winborn 
Slide Tape 
T. A. Ryan 

Slide Tape 
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COURSE OUTLINE 
(Monday, February 1, 1971) 



Mornlna 



8:00-8:10 
8:10-8:40 
8:40-8:45 

8:45-10:00 
10:00-10:20 
10:20-11:00 
11:00-11:55 

11:55-12:00 

Afternoon 

1:00-1:10 

1:10-1:30 

1:30-1:45. 

1:45-2:15 

2:15-2:30 
2:30-3:00 

3:00-3:20 
3:20-3:25 

3:25-4:55 
4:55-5:00 



Opening T. A. Ryan 

Evaluate Solution. to Problem B Staff 

Task Group Assignment; Problem C, 

Satellite T. A, Ryan 

Task Group Activity 

Recess 

Task Group Activity 



Evaluate Solutions to Problem C: 
Satellite 

Closing 



L. C. Silvern 



T. A. Ryan 
L. C. Silvern 



L. C. Silvern 



Opening 

Systems Using Synthesis and CAI 

Question and Ane^wer Period 

Reading and Interpreting Flowchart 
Models. Complex System Level 1: 
ABEC 

Question and Answer Period 

Reading and Interpreting Flowchart 
Models. Complex System. Level Z: 
Boeing 

Recess 

Task Group Assignment: Design a Model 
for Specific Setting 

Task Group Activity 

Closing T. A. Ryan 



L. C. Silvern 
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Morning 

8:00-8:10. 

8:19-8'>30 

8:30-10:00 

10:00-10:20 

10:20-11:50 

11:50-12:00 



COURSE OUTLINE 
(Tuesday, February 2, 1971) 



Opening 

Progress Reports 
Task Group Activity 
Recess 

Task Group Activity 
Closing 



T. A. Ryan 



T. A. Ryan 



Afternoon 

1:00-1:10 
1:10-3:00 
3:00-3:20 
3:20-4:50 
4:50-5:00 



) 



Opening 

Task Group Activity 
V Recess 



Evaluate Task Group Models. 
Closing 



T. A. Ryan 



Staff 

T. A. Ryan 
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COURSE OUTLINE 
(Wednesday, February 3, 1971) 



8:00-8:10 


Opening 


T. A. 


Ryan 


8:10-9:20 


Evaluate Task Group Models 


Staff 




9:20-.10:00 


Post Assessment 


T. A. 


Ryan 


10:00-10:20 


Recess 






10:20-10:40 


Program Evaluation 


T. A. 


Ryan 


10:40-11:30 


F'^nel: Implications of Systems 


D. G. 


Hays 






R. E. 


Hosford 




Ajpproach for Counseling, Guidance 


N. R. 


Stewart 




Counselor Education and 


B. B. 


Winborn 




Supervision 


L. C. 


silvern, Chairman 


11:30-11:45 


Open Discussion 






11:45-12:00 


Closing Remarks 


T. A. 


Ryan 
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APPENDIX G 



RESULTS OF PARTICIPANT EVALUATION 



Table I 

Group Profile of Mean Scores for Pre- and Posttest 
By Objective Component 



nut^^-j « ^ Mean Scores 

Objective Component Pretest PostteBt 





Systems concepts and principles 


15. 94 


17.75 




Develop, nonsense flowchart 


6.67 


12.06 




Interpret nonsense flowchart 


13.01 


15.31 




TOTAL 


36.42 


45. 12 



Table 2 



Performance Criterion Levels for Training Objectives and 
Level of Agreement with the Correct Answer by Objective Con5)onent 



Objective Component 


Percent of' Aftreemeat with Correct Amwer. 
Criterion Level Pretest Posttest 


Systems concepts and 


■ ■ ,86 


80 


89 


principles 




Develop nonsense flowchart 


80 


45 


80 


Interpret nonsense flowchart 


80 


82 


90 



MEAN 80 69 86 



APPENDIX G 



RESULTS OF PARTICIPANT EVALUATION 



Table 3 

Participant Self Evaluation Describing Perceived 
Attainment Level for Two Program Aims 



Percent of Participants Reaching Four Levels 
of Attainment of Aims Based on Self Evaluation 



Program Aim 




Very 


Quite 


Very Great 


No 




None 


Little 


a lot 


Amount 


Response 


Acquisition of new 












knowledge about 












aystms research 


0 


6 


59 


35 


0 


Development of profi- 












ciency in using systems 












techniques 


0 


6 


70 


18 


6 



^PENDIX H 

EVALUATION OF PROGRAM ORGANIZATION AND AOMINISXRATION 

Table 4 

Mean Rating of Training Program Learning Activities 



Mean Rating 
(Md • 3.28) 



Lectures 3.18 

Individual problems 3.89 

Progranmed instruction 3.06 

Task group assignments 3.24 

Gem^ral discussion 3.29 

Readings 3« 18 

Individual conferences with staff 3.82 

Audio-visual presentations 2.59 



Table 5 

Mean Rating of Training Program Instructional Materials 



Mean Rat ing 

References (Md • 3. 15) 



Banathy^ B. Instructional Systems. .3.00 

Hosford, R.E. and Ryan, T.A. Systems design 
in the development of counseling, and 

guidance programs. 3.31 

Hager, R.F. Preparing instructional objectives. 2.81 

Ryan, T.A. Systems techniiques for programs of 

counseling and counselor education. 3.38 

Silvern, L. C* Systems ent^lneerlng of education 
I: The evolution of systems thinking in 

education. 3. 18 

Silvern, L. C. LOGOS: A system lanjguage of 

flowchart modeling^ 3.24 



APPENDIX H 

EVALUATION OF PROGRAM ORGANIZATION AND ADMINISTRATION 



Table 6 
Mean Rating of Program Units 



Program Unit 



Mean Rating 
(Md - 3.30) 



Conceptualization of system in model form 3.29 

Model for producing a systems model 3,44 

Systems using feedback 3, 31 

Conceptual analysis and synth-^sis 3,26 

Rules and symbols for flowchart ipodeling 3.31 

Closed loop instructional system 3, 20 

Problem: Satellite Gonnunicat ion .3.12 

LOGOS workbook 3.06 

Nonsense problems 2, 94 

Problem: From Real Life Environment .3.88 

Illustrations of systems research, in counseling, 

testing, school negotiations, counselor education 3.50 
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EVALUATION OP PROGRAM ORGANIZATION AND ADMINISTRATION 

: Table 7 

Frequency of Responses on Rating of Program Management 



Management 
Component 



Item 
No. 



Description 



Frequency of Response 
on Four Levels of Agreement 
Strongly Dis- Strongly No 

Disa;;^ree Agree Agree Agree Response 



Program 
Information 



Pre-program information was 
adequate for my use in 
deciding whether or not to 
apply. 

Pre-program information 
accurately described the 







program offered. 


0 


3 


9 


4 


6 


Living 


3. 


Arrangements for meals and 












Accommo- 




living accpomodatioDs were 












dations 




satisfactory. 


1 


2 


12 


2 


5 


Staff 


4. 


Qualifications and compe- 












Competen* 




tencies of staff were 












cies 




satisfactory. , 


0 


n 
u 


7 


1 o 






5. 


The balance between formal 
















and Informal activities 
















was satisfactory. 


0 


4 


9 


4 


5 


Time 


6. 


There was sufficient time for 












Allocation 




individualized activities. 


0 


6 


6 


5 


5 


and 


7. 


There was sufficient time 












Utilization 




for group activities. 


0 


1 


9 


7 


s 

J 




8. 


There was suf f icieht time 
















for meeting informally with 
















other participants. 


1 


3 


9 


4 


5 




9. 


There was sufficient time 
















= for meeting with staff. 


0 


6 


6 


5 


5 




10. 


The daily schedule of activities 














was satisfactory (8:00-5:00). 


0 


2 


11 


3 


6 




11. 


There was opportunity for each 
















participant to express his 












Learning 


12. 


ideas and views . 


p 


3 


9 


5 


5 


Climate 


New acquaintances were made or 














renewed which will be helpful 
















in future professional work. 


0 


0 


12 


4 


6 




13. 


A presession on this topic 
















should be offered next year. 


0 


0 


5 


10 


7 




14. 


The scope and sequence of 
















learning experiences were 
















satisfactory. 


0 


1 


9 


6 


6 


Faciilties 


15. 


Physical arrangements (room, 














equipment, lighting) x^cre 












(1 




satisfactory. 


1 


4 


11 


0 


6 
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16. Ihe prograru met my expecta- 
tions 
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Final Report 



1971 AER A PRESESS I ON NUMBER ' F I VE ; EDUCATtONAL OBJECT \ VES : 
FORMULATION^ APPRAISAL AND ASSESSMEt^ 



General Goals . 

This presession was planned to develop skills in selecting, formulating, and 

APPRAISING THE OBJECTIVES OF SCHOOL PROGRAMS. In ADD I T ION, RELAX ED ISSUES, 

SUCH AS Natjonal. Assessment, performance contracting, and teacher accountability 
were considekeo, 

Faculty 

Directors: Eva L. Baker, University of California, Los Angeles; 

W. James Popham, University of California, Los Angeles. 

Staff: John D. McNeil, University of Cal ifornia, Los Angeles; 
Robert E. Stake, University of Illinois, Urqana; 
C. Mauri T2 Lindvall, University of Pittodurgh. 

Graduate Assistant: Arlene Gros$ Fink, University of California, Los Angeles. 
Object IVES ; 

1. Participants will be able to d ist inguish between instructional 
objectives with respect, to whether they do or do not possess the 

FOLLOWING characteristics:, 

A. MEASURAB IL ITY 

B. Student Minimal Level 

C. ClassMinimal Level 

D. Affect:ve, Psychomotor,.-or Cogn it i ve (Lowest Level or 

Higher than Lowest Level) Behavior 

E. Content Generality 

. 2. Given descriptions or affective measures, participants will be 

ABLE TO LABEL THEM CORRECTLY AS (a) DIRECT SELF-REPORT, {B) rhJFERENriAL 
SELF-REPORT, OR (c) OBSERVATIONAL INDICATORS. 
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3** Participants will be able to distinguish between procedures which 
cduld dr cduld ndt be used as a measure dp an instructional 
objective's attainment, for those which could $erve as criterion 
measures, the participant will be able to classify each according 
to a four category scheme involving thk following two dimensions: 
(l) Learner behavior versus product and (2) Natural versus manipulated 
conditiokis. 

Participants will be able to identify whether given instructional 

SITUATIONS, tests, DR MEASUREMENT PROCEDURES ARE MORE SUITED FOR 

criterion-referenced than fdr norm-r tferenc ed measurement. 
5* Participants will be able to indicate whether certain operations 
Conceivably usable in needs assessment procedures are associated 

WITH the needs assessment SCHEMES RECOMMENDED BY (a) STAKE, 

(b) Baker-Popham, (c) both, (d) neither. 

6. Participants will be able to identify, from multiple-choice 

ALTERNATIVES, S I TUAT I DNS ANo/oR OPERAT I DNS SUITABLE FDR ITEm/pERSON 

(matrix) sampling measurement procedures. 

7. Participants will display at least minimal knowledge of the special 
TOPICS (e.g., performance contracting, teacher competence assessment, 

AND BEHAVIOR ANALYSIS) TREATED DURING THE PRESESSIDN BY CORRECTLY 
ANSWERING MULTIPLE-CHOICE QUESTIONS DEALING WITH THE KEY ASPECTS OF 

* these topics. " 

8. Participants will be able to alter given objectives so that they 

POSSESS ANY OF THE CHARACTERISTICS (A THROUGH E) DESIGNATED IN 

Object i ve 1 above. 

9. Given general instructional goals, both cognitive and noncognit ive, 
participants will be able to generate a wide variety of measurable 

. instructional objectives which might be employed to operat i onal i 2e 
such goals. 
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10. Participants will be able to develop a simple behavior analysis 
given a desired terminal learner behavior. 

11. Given fictitious descriptions of educational personnel engaged in 

THE FORMULATION, APPRAISAL, ANo/oR ASSESSMENT OF INSTRUCTIONAL 
OBJECTIVES, PARTICIPANTS WILL BE ABLE TO IDENTIFY SPECIFIC PRACTICES 
NOT CONSONANT WITH PROCEDURES RECOMMENDED DURING THE PRESESSION. 

l2t Participants will be able to prepare drafts of item forms which include 

•] the following attributes: task DEFINltlON> CONTENT LIMITS, ITEM V"0RMAT, 

y • • " . * ■ 

I CRITERIA, AND SAMPLE ITEMS. 

5 ■ " ■ ■ 

\ procedure ! 

\ Prior to the presession, the part ic ipants rece i ved a statement of general and 

\ SPECIFIC GOALS AS WELL AS A SCHEDULE INCLUDING TOPICS AND INSTRUCTORS. 

I Mater lALs : . 

k ■ 
\ Major materials distributed; . 

> - ■ • 

\ ' ■ A) Samples of the Instructional Objectives Exchange Collections, e.g> 
% Attitude Toward Learning , Measures of Self-Concept . 

I These collections include instructional object ives . and relevant 

\ ■ ■ 

% TEST items. 

^" ■ ■ . • . 

\ B) WR ITTEN PRACTICE exercises 

I C) Simulation exercises 

i D) Selected bibliography 

5 ■ . . ■ • 

i E) Reprints of data pBTAiNED while working with instructional objectives 

I F) Reprints of journal articles 

? ' •-. 

f' ■ ■ ■ 

t ' ■ . - * 

i Participants: 

^ mm - 

I Of the ninety-one participants, approximately HAD UNIVERSITY-AFFILIATED 

I • ■ . ' "... 

I POSITIONS, 30J^.WERE RESEARCH CONSULTANTS OR CO-ORO I NATORS, AND THE REMAINDER 

'i\ - • ' . .... 

I WERE DEANS, PRINCIPALS OR SUPERVISORS IN PUBLIC AND PRIVATE SCHOOLS. 

I • .. ■ ■ • ■ 
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Schedule 

Saturday, January 3^5 

The morning was devoted to pretesting and to discussion of the historical 
evolutionof objectives and a review of basic concepts, part i c i pants^ tests were 
immediately scored, results discussed and modifications in the schedule were 

MADE, 

In THE afternoon, affective goals and NGN-reactive measures were analyzed. 
The evening session was an optional informal discussion session. While 
practice sessions were planned for the evening, the high pretest performance 

suggested THAT AN OPEN DISCUSSION WOULD BE MORE APPROPRIATE. 

Sunday , January 31; 

In the MORNING, CRITERION-REFERENCED MEASUREMENT, ITEM FORMS AND ITEm/pERSON 
SAMPLING WERE TREATED. ' 

The AFTERNOO?i SESSION WAS .CEVOTED TO SIMULATION SESSIONS IN WHICH PARTICIPANTS 
DEVISED OBJECTIVES AND MEASURES, AND HA? THEIR PRODUCTS REVIEWED. 

The evening TOP I c was concerned with the role of objectives in teacher 
competence assessment and instructonal supervision. 
Monday , FebruaryI; 

Educational needs assessment and the establishment of priorities among 
competing objectives constituted the bulk of the morning discussions. 

The afternoon session involved an analysis of objectives and the means 
by which they can be sequenced to optimize individually prescribed instruction. 

The afternoon session culminated in a cocktail party, sponsored by the 
instructonal objectives exchange, and which included presession participants, 
staff, as well as staff from other presessions. 

In the EVENING, NATIONAL ASSESSMENT WAS DISCUSSED AND ANALYZED. 

Tuesday , February 2: , 

The role of objectives in contemporary evaluation models and the use of 
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objectives in formative and summat i v e evaluat i on weir e olscusseo in the morning. 

The use of objectives in developing lesson materials was discussed in the 
afternoon session. 

Lack of room availability precluded an evening session. 
Wednesday ^ February 3: 

The morning was. devoted to a discussion of needed research. The presession 
was summarized, and the participants received the posttest. 

The posTTEST results were returned in the afternoon. 

Evaluations 

1) THH participants RECEIVED A PRETEST AND A POSTTEST. ThE RESULTS BY 
OBJECT I VE were: 

Objective . Pretest Posttest 

lA 8ojg 91^ 

IB 75 • 92 

IC 70 7^ 

ID 03 94 

IE 7? 79 

^ 2 . 7^ . 75 

. i 69 • 87 

6 27 67 

7 39 66 

Objectives 8 through l2 were evaluated as follows: An identical item for 
each objective was used in the pretest and postt est , s i nc e all. items 
reflected constructed rather than selected responses. participants were 
asked to identify their tests by using their mot her * s ma 1 0 en name, 
a small sample of test papers were' then randomly selected and paired so 
that. a pretest and posttest was available for each of ^^lo participants. 
These forms were coded ("pretest* or posttest") by the presession graduate 
assistant, then given to the staff member responsible for each ob j ect i ve 

WHO, WITHOUT KNOWING WHICH WAS PRETEST OR POSTTEST^ SCORED ONE PAPER IN 
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each pair as super lor* for all five objectives ^ 80^ or better of the 
posttest responses were scored as super i or ♦ 
2) The evaluation form completed by the participants was the official form 

GIVEN TO ALL l^fX PRCSESSIONS« 

The results were summarized as follows--- 




/ERA 1371 'Research Training Sessions 
Evaluation by Participants 



(Name of Session; objectives 
I Director: pop ham 



I lA. To what extent did the relative availability or unavailability of books 
. and journals interfere with or promote your attempts to master the con- 
tent of this session? 

. NON E LITTLE . 29^ SOME ' 

IB. To what ex-tent did reproduced materials given to "you by the staff improve 
matters? 

HELPFU L 62^; VERY HELPFUL gfi,^ . 



I 2A. Did you feel that you lacked a "place to work", either alone or in small 
groups? 



YES__Jj_^__^ N O ^15: No COMMENT 8^ 



I 28. Was your room satisfactory? 

• YES he% N O k6% No COMMENT 81 



I 3A. Which features of the meeting rooms wara inadequate or not conducive to 



learning? 



BLACKBOARDS ■ LACK OF OVERHEAD 

PROJECTOR 

sizE__ . ;. ■ - 

FURNISHINGS 
LIGHT 



. ■ MISCELLANEOUS_; 

AIR v/ (rn.n) ' ' 

I 38. Which features were especially facilitative In the same regard? 

$ . ' 

LIGHT . AIR 



SOUN D • FURNISH I NGS_ 

CONVENIENCE No COMMENT_ 
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J 4A. Was five days too long a period to' leave your work at home for the purpose 
of attending this session? 



YES 32^ N O 68^ 



I AB. Was five days too short a period in which to learn much of the content of 
^ this session? 

I ' . YES 2^ . NO 



I 5A. Were you allowed enough time in which, to pursue activities of .your own 

? choosi ng? 

I YES N O 27^ No COMMEN T. iH 

I 5B. Would you have preferred not, to meet in the evening after dinner? 

I YES 22^ N O k'^ No COMMENT f 



I 5C. Would you have preferred more of^^ fewer meetings per day than there actually 
■;: were or was the number of meetings per day agreeable to you? . 

I ■ ■ . • * .* 

I ' ■ . FEWE R . ENOUG H 90$g . MORE 



I 6A. Were the'indi vldual lectures too long to sit and listen or take notes? 



YES 5^ : NO _93i 



!i ■ . . ' . ■ . . . 

5 6B. Were the lectures scheduled In. an appropriate sequence? 
1 YE S fe^ NO__35^__ 



Did you have sufficient opportunities to Interact with other participants? 
YES NO^ : 



3 

I 8A. Were the instructors too inaccessible or unapproachable so that you did not 
i get the Individual attention that you desired? 

t . • " " ■ ■ 

i YES -y.s^: N O 87,^^ No COMMEN T c^eg 



I 86. Was it helpful to- have graduate student assistants present? 

I ERiC YES 57^ NO . 5^ No COMMEN T ^8«g 
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16. Do you feel that AERA is making an important contribution to education 
by sponsoring sessions such as this one? 

YES lOO^ ^WnO_ • 

17. Do you feel that anything has happened during these five days to make it 
more likely that you wl I I leave your present position of employment? 

YES Yn;^: V . 

18. Is It Ukely that you will collaborate in research with someone else 
attending this session (other than those you already were likely to 
col laborate with)? " 

YES . N O (M 

19. Do you think that the staf f shoul d feel that It has accomplished its ob- 
jectives during this five-day session? 

• YES. ■ ■ NO 9^ 1 



Note: The participants felt that the room was too colo and poorly ventilated. 
(Items 3A and 36) 



Director* s Evaluat ion: 

The pftEscsoiON was generally successful in terms of its stated objectives. 
However, as expected in a large group the variability in participants' 
experience with the topic caused the staff some problems. Small group sessions 
appeared to be a partial solution and in addition enabled the participants to 
become very well acquainted. certainly the sophistication of instructional 
procedures employed would have been increased had there been opportunity for 
staff planning meetings. 

Yet the session did not consist, in any major way, of presestat ions of previous 

RESEARCH WORK CONDUCTED BY THE STAFF. ThERE WAS A CLEAR INTENTION TO MAKE THE 
SESSION A TRAIN ING ENTERPRISE WHERE SOME TANGIBLE SKILLS WERE TRANSMITTED IN 
ADDITION TO FREE TRANSFER OF IDEAS ABOUT THE TOPIC. 

The PRESESSION seemed to be CHARACTERIZED BY A SiPIRIT OF GOODWILL AND OPEN 
EXCHANGE AMONG STAFF AND PARTICIPANTS, ThERE WAS RARELY UNANIMITY BY STAFF 
OF OPINION ON ANY BUT THE MOST MUNDANE CONCERNS. THE STAFF PROF ITED A GREAT 
DEAL FROM THE INTENSE INTERACTION REQUIRED IN A SHORT COURSE* ' CaCH OF US HAS 
RECEIVED LETTERS FROM THE PARTICIPANTS PURSUING POINTS. RA ISED OUR I NG THE 
PRESESSION. 



AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 
1971 Research Training Session 
Educational ol)jectives: Formulation, Appraisal & Assessment 

PARTICIPANT ROSTER 



1. Mildred- Bain 

Miami-Dade Jr. College, 
11011 SW 104 Street 
Miami, Florida 33156 

2 • William* Beavers 

Illinois State University 
Department of Education 
Normal, Illinois 

3. Dr. Bernsten 

Bureau of Educational Research 
University of North Dakota 
Grand Forks, North Dakota 

4. George Bevan 

3205 Lakehill Cres. 
School District #51 
Lethbridge, Alberto 
Canada 

5. Suzanne Blackman 
2601-51 Soldiers Home Road 
W. Lafayette, Indiana 

6. Marjorie Boeck 
ORME 

E. 313 Health Sciences 
University of Washington 
Seattle, Washington 

7. Roscoe A. Boyer 

Bureau of School Services 
University of Mississippi 
University, Mississippi 

8. Henry Campiglia 
Research for Better Schools 
1700 Market Street 

Penn State University 
Philadelphia, Pennsylvania 



9. Reuben Chapman 
c/o Baum 
360 E. 65th 
New York, New York 

10. Lester Clark 
Office of Planning 
Te?cas Educational Agency 
Austin, Texas 

11. John Cholvat 
150 Akburn Cres. 
Willodale, Ontario 
Canada 

12. Ruber t Cullen 

Office of Res. in Medical 

Education 
University of Washington 
Seattle, Washington 

13. William Gulp 
Gannon College 
Erie, Pennsylvania 

14. Deborah Diamond 
15 Overhill Road 
Scarsdale, New York 

15. Del Eberhardt 

40 Hendrie Avenue 
Riverside, Conn. 

16. Clifford Edwards 
Education Department 
Illinois State University 
Normal , Illinois 
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17. Judith Eisler 
CUE 

105 Madison Drive 
New York, New York 

18. Ijouire Fisher 
Miami-Dade Jr. College 
11830 NW 27 Avenue 
Miami, Florida 

19. Christopher Flizak 
Department of Public Instruction 
126 Langdon Street 
Madison, Wisconsin 

20. Ernest Bent ley 
Atlanta Public Schools 
90A Mony Building 

1655 Peachtree Street NE 
Atlanta, Georgia 

21. Franklin Greenough 

E. 313 Health Sciences Bldg. 
University of Washington 
Seattle, Washington 

22. Thomas Gould 

12 North Hill Road 
Balls ton Lake, New York 

23. Joe Hannabach 

Research for Better Schools 
1700 Market Street 
Philadelphia, Pennsylvania 

2A. Russell Hill 

Research for Better Schools 
1700 Market Street 
Philadelphia , Pennsylvania 

25. Mary Hoaglund 

39 Evergreen Circle 
Princeton, New Jersey 

26. Irvin Hochman 
120 summit Avenue 
Dumont, New Jersey 

27. E.J.F. Hodgett 
28 Tupper Street 
St. John's 

Newfoundland , Canada 



28. Dorothy Horn 
OISE 

252 Bloor Street 
Toronto, Ontario, Canada 

29. Milton Houpt 

c/o New Jersey College of 

Dentistry 
201 Cornel is on Avenue 
Jersey City, New Jersey 

30. Jack Hutton, Jr. 
Association of American 

Medical Colleges 
1 Dupont Circule, N.W. , Suite 200 
Washington, D.C. ' 

31. Vera Ireland 

25 5 A Preach tree Road, N.W. 
'Atlanta, Georgia 

32. Patricia Johnspn 

Office of Research in Education 
Tufts Medical School 
Harrison Avenue 
Boston, Massachusetts 

33. Osman Kazanci 

1570 G. Spartan Village 
East Lansing, Michigan 

3A. G.P. Killian 
Northwest REL 
710 S.W. 2rid Avertue 
Portland, Oregon 

35. Robert Klein 

Division of Community Services 
Montclair State College 
Upper Montclair, New J. 

36. Kathleen Kochuba 
ORE 

Tufts Medical School 
Harrision Avenue 
Boston, Massachusetts 

37. Leonard Kreit 
Dental Health Center 
Educational Development Br. 
14th Aven. and Lake Street 
San Francisco. Calif. 



38. Shirley Kreutz 
2501 N. Street 
Lincoln, Nebraska 

39. Molly Laplalne 

Bureau of Staff Development 

and Training 
Towar A, Place-de-Ville 
Ottawa, Ontario, Canada 

40. William Lawton 
Rhode Island College 
Providence, Rhode Island 

41. Jim Lewis 
Department of Education 
Box 911 

Harrisburg, Fenn. 

42. Carroll Liechti 
Wichita Public Schools 
428 S. Broadway 
Wichita, Kansas 

43. Carl Lindsay 
One Shields . 

Penn State University 
University Park, Penn. 

44. Lawrence Lipton 

School District of Phil. 
Simon Gratz High School 
18th and Hunting Park 
Phil., Pa. 

45. Stanley Lisser 
CUE 

105 Madison Avenue 
New York, New York 

46. Mary Lydon 

Pa. Bureau of Ed. Res. 
Box 911 

Harrisburg, ta. 

47 . Alan Maher 

340 Euclide Avenue 
Massapequa Park, N.Y.. 

48. Mertin Manley 

Dept. of Ed. Foundation 

wsu 

^ Whitewater, Wise. 
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49. Floyd McKinney 
152 Taylor 

University of Kentucky 
Lexington, Kentucky 

50. Mary Meehan 

806 Santa Fe Trail 
Kansas City, Mo. 

51. Robert Nearine 
Board of Education 
249 High Street 
Hartford, Con. 

52. Ervin Neff 

Minnesota Higher Ed. Comm. 
550 Cedar Street Suite 400 
Capitol Square Building 
St. SPaul, Minn. 

53. Scott Newcomb 

Nat. Assessmt. of Ed. Progress 
2222 Fuller Road 
Ann Arbor, Michigan 

54. James Nighswander 
2430 Westchester Blvd. 
Springfield, 111. 

55. Walter Pierce 
1108 E. Grove 
Bloomington, Illinois 

56. Joseph Pietropaolo 
1564 Payne Road 
Caxiandaigua, New York 

57. Fred Pigge 
Dept. of Ed. 

Bowling Green State Un. 
Bowling Green, Ohio 

58. Robert Pinney 
UMREL 

1640 3. 78th Street 
Minn. , Minn. 

59. Charles Planz 

172 Genesee Street 
Rochester, New York' 

60. Richard Poole 
206 Cap en Blvd. 

. Buffalo , New York 



Jim Poteet 

Layfayette School Corporation 
Lafayette, Indiana 

Terry Puckett 

Memphis State Technical Institute 
5983 Macon Cove 
Memphis, Tenn. 

David Quist 

5 Surburban Road 

Worcester, Mass. 

Helen Randolph 

195 Willoughby Drive 

Brooklyn, New York 

Fred Rivkin 
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PRESESSION VII 



1. Title: The Psychology of Written Instruction 



.2. Staff: Ernst Z. Rothkopf 
(Ccdirector) 



Bell Laboratories 
Murray Hill, Mev/ Jersey 



Lawrence T. Prase 
.( Codirector) .. 



Bell Laboratories 
Murray Hill, Nev; Jersey 



Barbara Musgrave 



Smith College 
Northampton, Massachusetts 



Paul Johnson 



University of Minnesota' 
Minneapolis, Minnesota 



J. Douglas Carroll 



Bell Laboratories 
Murray Hill, New Jersey 



Edward Crothers 



University of Colorado 
Boulder, Colorado 



George Klare 



Ohio University 
Athens, Ohio 



Wells Hively, III 



Robert Calfee 



University of Minnesota 
Minneapolis, Minnesota 



Stanford University 
Stanford , California 



3. General Description: 

The stimulus for this presession was the recent 
growth of interest and research in prose learning. TMs 
is an especially important topic in the study of . . 



instructional processes from both a theoretical and 
practical standpoint • The issues involved in written 
instruction are related to verbal problem solving and. 
thinking in general and especially to problems in the 
rational improvement of documentary instructional 
systems • * • 

Our intention was to bring together a small 
number of researchers for an intensive five days of 
lectures, each followed by small-group discussions of 
the topics covered in the lectures. Participants also 
were asked to bring some problem or research plan of their 
own, which provided another focal point for the small- 
group discussions. 

It was clear from the outset that a great deal 

of content and discussion v^ould have to be contained 

within the five days alloted to the presession. Three 

decisions greatly facilitated accomplishing this task. 

First, we selected Columbia University's Greyston House 
I 

(^n the Riverdale sectior^ .pf the Bronx) as a relatively 
secluded retreat. All but a few of the participants 
thus remained a;board for meals and discussions that 
sometimes lasted into the early morning hours. Second, 
articles suggested by the guest speakers . were mailed 
to participants before the presession. Thus, more time 
could be devoted to discussion, participants were pre- • 
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pared in advance for sp'eclflc topics, and the coverage 
of our own In-house library could .be increased. A 
third decision that Increased the effectiveness of the 
presesslon in terms of individual attention, was our 
decision to limit the number of participants to somev;here 
around twenty. In this way, we were able to cover, in 
depth, the research plans generated by each participant. 

Planning, preparation of materials, contacting 
speakers, arranging accommodations, etc., involved about 
50 hours of labor. Another I6 hours of preparation for 
all speakers brings this total to 66 hours of planning 
activities. 

4, Objectives: 

We felt that a presesslon on thlS' topic could 
accomplish a variety of objectives, among which were the 
following. First, to provide a framework within which 
participants could conpeptuallze major problems in this 
relatively new area of research. The presesslon 
announcement, published in the Educational Researcher , 
described the presesslon in terms of' this framework 
which divided the psychology of -written instruction into 
four topical areasj (l) analysis of content, (2) textual 
representation of content, (3) models of the reading 
process, and (4) measurement of learning outcomes. Our 
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decisions concerning tKis framework resulted from several 
brief meetings culminating in this conceptual analysis 
of the field. Decisions about selecting the staff were 
easily made once this framework had been established. . 

A second objective was to provide participants 
with theoretical perspectives in each of these topical 
areas. A third objective was to provide participants 
with usable techniques related to these theoretical ideas. 
A fourth objective was to give participants an opportunity 
to discuss theoretical ideas with their/proponents, and 
to use associated techniques in some realistic way. Our 
decision to invite several guest speakers arose,, not 
only from our concern for adequate coverage, but from 
the excessive demands that we anticipated if only two 
or three people were to conduct such a session. 
5. Schedule: 

First Day: Morning: 

Introductory comments, general concepts 
(Rothkopf, Prase, Musgrave) . 

Afternoon: 

Analysis of^^^rfstructional content; multi- 
dimensional scaling (Carroll and Johnson) « 
Practicum (Staff). 



Second Day: Morning: 

Analysis of instructional content; 
associative structure and digraph analysis 
(Johnson and Prase). 
•AfternoVn: 

Textual representation and problems in 
psychollngulstlcs (Crothers). Practicum (Staf 

Third Day: Morning:: 

Textual representation and information 
theory (Musgrave). 

Afternoon: 

Readability (Klare). Practicum (Staff). 

Fourth Day: Morning: 

Experimental control of study activities 
(Rothkopf, Prase) . 

Afternoon: 

Models of the reading process (Calfes). 
Practicum in research techniques (Staff). 

Fifth Day: Morning: 

Measurement of learning outcomes (Hively) .• 

Afternoon: 

Practicum in the design of experiments on 
written instruction, evaluation questionnaires 
summary of presession (Rothkopf, Prase, Mussra 
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6. Participants: 

Prom a total of 38 applications, 25 participants 
were Selected on the basis of the Information contained, 
on the pre session application. V/e felt that this 
Information was- Inadequate and that some provision should 
be made to Increase the flexibility of the application 
forms, perhaps, by Including a number of queries that 
-could be -optional for different presesslons. The relevance 
of these Items could be stated briefly In the announcement 
of each presession. Evidently, our criteria for selection, 
which consisted of membership In certain professional 
associations, amount of time devoted to -research, and . 
relevance of the presession to the participants' work, 
were fairly effective. On a questionnaire administered 
after the presession, 94j^ of the respondents indicated 
that in their opinion, participants . had been selected 
appropriately. With a small number of additional application 
questions we feel that this number could be 100^. 

Of the total number of 22 participants who 
finally attended, 64$^ were male and 36^ female. Mean 
age was. 35. 2 years. The average year of highest degree 
was between 1967-I968 (whethq^r a mean, median or mode 
is used as an average). Eighty-six percent of the 
participants held the Ph.D. degree, which was in line 



with our presesslon anrlouncement. Participants devote 
a mean pf 58,1?^ of their time to research, with a mean 
number of 4,59 publications. Favorable participant 
response to the presession suggests that this group 
composition was appropriate for our purposes. 

Participants came from all over the U.S,Aa 



and from Canada • About 70J^ 



came from the east coast 



and the midwest. The remainder came from the southv/e:: 
southeast, and^wte^ coast, 

7, Instructional Material: 

Readings vxere suggested by the staff before 
presession began. The most critical articles were mai 
tq participants before th^ presession. Books and 
secondary articles (about 10 books arid' thirty article^: 
provided the content for our in-house library. In 
addition, exemplary experirpental materials were kept i 
the library, Reproductlon^facilities were also availj, 
on the premises in. case participants needed them. 

8. Evaluation: 

On the last dky ;oX the presession, particl;:!^ 
filled out two questionnaires; one supplied by AERA an 
another of our own design. Our own questionnaire ihcl 
essay, multiple choice and semantic differential itern.: 

> Accommodations , — Figure 1, on the next pag. 
summarizes the semantic differential data. Eighteen o 
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the questionnaires were usable. On our questionnaire, 
88^ of the respondents Indicated that the ^accommodations 
were adequate, v/hlch agrees, with the average ratings of 
Items 9 and 10 in Pig. 1. In an essay question asking 
for!?,thii most favorable characteristic of the presession, 
the most frequent response was the Interpersonal interaction 
that evolved during the five days. Participants attributed 
this to the fact that the lodgings were isolated from the 
busy downtown area. Since the staff remained at Greyston, 
it is not surprising that 93^ of the respondents disagreed • 
with the statement that the staff was too inaccesslblev 
The cloistered arrangement seems optimal for our purposes; 
8l^ of the respondents said that they anticipated maintaining 
some contact with the staff, and 82% felt that they had 
sufficient opportunity to Interact with other, participants. 
The fev/ exceptions were those whoj. for one reason or 
another, commuted from the New York area and thus missed 
evening discussions. Twenty-nine percent of the 
respondents would have preferred not to meet in the 
evening. 

I 

Fifty-Dollar Fee .— About 35^ of the respondents 
indicated that the $50 fee was a financial burden. For 
others, this fee was paid by their home Institution. 

Lecture-Discussion . — Items 2, 4, 5 and 6 In 
Plgi 1 Indicate that the resppndents felt that more time 



should have been devoted to small-group discussion. During 
the day^ there were lectures each morning and afternoon. 
Although 71^ of the respondents felt that the lectures 
were not too long to sit through (they were broken by 
coffee breaks), the pace was quite demanding. In the 
future. It might be well to hold srifiall-group discussions 
during, the day and to make speakers available to participant 
on a less formal basis. Small-group discussions were 
seen as extremely useful (Figure l) . It Is precisely 
these discussions that were missed, by commuters. Hence, 
discussions should be conducted during the day, or all 
participants should be required, to attend evening sessions. 

Aside from interpersonal Interaction, the 
respondents saw theoretical models and research Ideas as 
an Important outcome from the presesslon (essay data). 
The most useful te^chnlques acquired Included theoretical 
perspectives, readability and scaling methodology. All 
respondents (100^) said that they had. acquired some useful 
techniques . or research Ideas from the presesslon, although 
only ^3fo said that the content wais what, they had expected^ 
Perhaps there Is a nee.d for a more complete description 
of presesslon contents in The Educational Researcher . 

General Comments .-- About 8l^ of the respon^^'ents 
disagreed with the statement that the presesslon v/as too 
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longj 88?^ disagreed with the statement that it was too 
short. The duration appeared to be about right. All 
respondents (100,^) felt that the timing of the precession 
(immediately before the annual AERA meeting) was 
appropriate. ' In addition, 94^ of the respondents felt 
that AERA was making an important contribution by- 
sponsoring these sessions. The same percentage felt that 
the staff should feel that it had accomplished its 
objectives. Eighty-tv/o percent felt that another pre- 
session on the psychology of written instruction should 
be held next year. 
9. Directors' Evaluation: 

Responses of participants, comments gathered 
at the Annual AERA Meeting/ and our impressions suggest, 
that the presession accomplished its objectives. 
Participants came away from the presession with usable 
research techniques and specific experimental ideas, as 
well as a broad view of the problems and prospects . for 
research in this area. 

In. particular, we suggest that a retreat-like 
atmosphere makes it possible to accomplish a great deal 
in five days. Evening special-interest small-group 
discussions might; be effective if they begin the second 
day of the session, although we waited until the third 
day v;hen participants' In-yerests had .become firmly 
established to formalize these meetings. , 



A good deal of planning and preparat'ion time 
was expended. Much time (and .materials) was contributed 
by the home institutions of the speakers and codirectors 
and was not represented in our financial report. In 
particular^ we felt that additional money should be 
provided for speakers and graduate assistants. 
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I. Presesslon Objectives 

Our purpose in teaching this five day presesslon was to present the cost 
current and potentially most useful nonpar ame trie statistical procedures that 
could be applied in educational research. It was our hope that the researchers 
who attended this session would be able to put these techniques to use in their 
own research and could guide others in the application of the techniques. Since 
nonparametric methods are increasing in number and in versatility of application 
it is essential that active educators and behavioral researchers be aware of the 
advances in this important area of research jnethodology . 

A printed statement of the objectives of the presesslon was made available 
to prospective participants in the November 1970 announcement of the AERA Re- 
search Training Programs which appeared in the Educational Researcher . The 
specific announcement read as follows: 

Content and Objectives: The purpose of this presesslon will be to 
increase the competency of the behavioral scientist in the design and* 
analysis of experiments employing qualitative and quantitative variables. 
The presesslon will have a dual emphasis: 

1. the extension of contingency table techniques to complex 
design* testing, and estimation problems in educational 
research. 

2. the introduction of powerful statistical procedures which 
are less restrictive in their assumptions than are classical 
procedures and whicch (sic) are applicable to a wide variety 
of problems in data analysis. 

It is hoped that exposure to these procedures will aid the participant 
in solving problems dealing with experimental design, hypothesis testing, 
and estimation commonly found in educational research. 

The content of .the course will be divided into three parts: rank 
procedures to test, for location, contingency table procedures , for 
qualitative data, and techniques for measuring association. To empliasize 
the parametric-nonparametric analogies, we plan to begin instruction 
with the rank procedures, r . 
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Anticipated Audience: This session will be open to holders of a 
doctorate or doctoral candldiates in education and allied fields whose 
academic responsibilities include or will include the design of 
educational research studies and analysis of research data. The course 
is intended for educational researchers whose primary commitment is to 
substantive areas other than statistics and measurement. Participants 
will be expected to have a basic knowledge of inferential statistics. 
Generallly, this will imply a familiarity with the basic elementary 
statistical techniques usually presented in a two quarter or two 
semester course in statistics. 

Our Intent was to plan each session so that there would be adequate time 
to cover the topic at hand with a minimum of stress on the participants. This 
was handled by scheduling three ninety minute sessions each day with a sixty 
minute informal discussion period at the conclusion of the day's lectures. We 
restricted ourselves to those nonparametric techniques which we believed to be 
potentially the most Important and useful to researchers in education and the 
behavioral sciences. We realized that researchers, no matter how diligent, 
could not^ achieve complete mastery of these techniques within the limited time 
period of the presesslon. Therefore we tried to introduce our participants to 
a wide variety of techniques during the presesslon and to provide them with 
extensive supporting materials to enable them to continue their independent 
study in nonparametric methods when the presesslon was completed. Our previous 
experience with the presesslon participants of 1968 > 1969, and 1970 gave us 
reason to believe that the participants would continue their study of the topics 
a^ their use of the presesslon materials long after the conclusion of the 
presesslon. 

In all cases the discussion and the use of tests and confidence intervals 
focused on applications. The mathematical development and proving of theorems 
and related results were not covered as a part of the formal lecture periods. 
These theoretical developments appeared as separate sections in the handout 
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materials distributed to the presesslon participants. 

Although this presesslon was primarily concerned with nonpar ametric statisti- 
cal techniques for the analysis of qualitative and quantitative variables, it 
was recognized that experimental design considerations would affect the parti- 
cipants' ability: 

(a) to provide interpretable estimates of treatment effects; 

(b) ' to employ precise statistical techniques in data analysis. 

For this reason, the informal discussion periods were used for more extensive 
consideration of the design problems associated with the statistical techniques 
presented. To stimulate discussion of research design considerations and of 
nonparametric analysis procedures, many detailed examples of studies requiring 
a nonparametric analysis were presented as separate handouts. 

It was hoped that exposure to these procedures would aid the participant in 
solving problems dealing with experimental design, hypothesis testing, and 
estimation commonly found in educational research. Specifically, it was our 
expectation that, as a result of participation in the presesslon and subsequent 
independent study of the presesslon materials, the participants would be able to: 

(a) Explain the rationale behind the test procedures presented. 

(b) Perform these tests on data from educational research. 

(c) Apply and extend the techniques learned to specific problems 
in educational research. 

(d) Distinguish between experimental designs in which variables are con- 
founded and designs which provide interpretable estimates of treat- 
ment effects. 

(e) En^loy "precision" as one criterion in selecting an appropriate experi- 
mental design and the "best" statistical test for a specific situation. 

Ifi (f) Read current literature involving methods deseribed in this prcsession. 

(g) Direct other researchers in the use of these statistical techni qucMi. 
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The content of the course was divided into three parts: rank procedures 
to te»t for location, contingency table procedures for qualitative data, and 
techniques for measuring association. In previous years » the presession on 
nonparametric methods has started with a consideration of the techniques appro- 
priate for the analysis of qualitative data. While this approach has had the 
advantage of linking the nonparametric methods closely to their probabilistic 
models, it has meant that instruction has begun witl:\ the technique which is 
probably least obvious to the participants and that initial instruction has 
not made use of the participants' familiarity with parametric statistics. To 
avoid these difficulties and to emphasize the parametric-nonparametric analogues 
we began instruction with the rank procedures. 

Part I: The initial lecture reviewed basic probabilistic concepts and simple 
analysis of variance designs. A rationale for the use of nonparametric tests 
was presented. Tests and interval estimation procedures based on .ranks were 
introduced and were compared with their parametric analogues. These Included 
the one~sample Wilcoxon Matched-pairs test, the Mann-Whitney test, the Kruskal- 
Wallis test, the Friedman test and the corresponding confidence interval pro- 
cedures. Extensions of the rank and normal scores procedures to more complex 
design problems were considered. 

Part II: The lectures of this section dealt with tests and interval estimation 
procedures designed to handle qualitative variables. These included the Irwin- 
Fisher Exact test, the Median test, and the Chi-square test of Homogeneity. The 
familiar chi-square tests of association and equality of proportions were 
extended to handle the multi-variable problems that commonly occur in survey re- 
search. Post hoc procedures using the chi-square distribution were also introdu 
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Part III: The last part of the course dealt with measures of association 



for both qualitative and quantitative variables, e.g., the Chi-square test of 



Independence, the mean-square contingency coefficient, tests for monotonicity 



based on Kendall's tau, Kendall' 



s coefficient of concordance and Spearman's 



rho and Kendall's tau. 



II. Staff 



The staff consisted of two co-directors, two instructors, and one graduate 



assistant. Illness prevented the attendance of one of the instructors; however, 



he contributed substantially to the preparation of materials for the presession. 



Staff members were: 



Co-directors : 



Maryellen McSweeney 



Associate Professor of Education 
464 Erickson Hall 
Michigan State University 
East Lansing, Michigan 



Andrew C. Porter 

Associate Professor of Education 
201 D Erickson Hall 
Michigan State University 
East Lansing, Michigan 



Associate Professor of Education 
4511 Tolman Hall 

University of California, Berkeley 
David J. Wright 

National Assessment of Educational Progress 
Ann Arbor, Michigan 



Office of Research Consultation 
Michigan State University 



Instructors : 



Leonard A. Marascuilo 



Graduate 
Assistant : 



James Maas 



> 
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All individuals on the staff have a deep interest in the area of non- 
parametric statistics. They are proven teachers equipped with a thorough 
knowledge of distribution free statistical methods and research design. 

Despite the geographic separation of the staff, members of the instructional 
staff shared similar interests in and knowledge of nonparametric statistics 
and had collaborated on the planning and coordination of the presession. Ex- 
tensive work on the preparation of presession materials involved all of the 
staff members prior to the actual session. During the presession the staff 
members were available to the participants on an informal basis to discuss 
'individual research projects, answer questions and just generally "talk shop." 
The staff members worked exceedingly hard and were well-rewarded by the 
positive responses of the participants. 

III. Selection and Notification of the Participants 

The eligibility of each of the- applicants was jointly assessed by the 
co-directors. A few applicants were rejected on the basis of their inadequate 
preparation in parametric statistical techniques (Entry skills at the level 
of . sophistication of a two-quarter course in inferential statistics were 
required). Later the over-subscription of the presession forced us to reject 
adclltlonal applicants. All persons whose applications were received prior to 

0.., 

January 27, 1971 were. notified immediately of their acceptance or rejection. 

A form letter of acceptance was' sent to the intended participants, together 
with an outline of the objectives and content of the presession and a summary 
of probabilistic concepts to be used in the presession. Additional materials 
pertaining to hotel registration and AERA reimbursement were also enclosed. 
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IV, .Description of the Participants 

A total of 54 Individuals were accepted for this presesslon, but 5 
notified us qf their inability to attend prior to the presesslon, 3 did not 
notify and did not attend, and 1 attended without having applied to the 
co-directors. Thus, a total of 47 participants were in attendance for this 
presesslon. • * 

The 47 participants are listed and described below in a summary of the 
biographical inforraatlori given on their application forms; 



Sami J. A lam — . . , 

509 Stanton 

Port Huron, Michigan 

Linda Bilsky 

Box 89 ' 
Teachers College 
Columbia University . , 

New York, New York 

Jack H. Bond 
Evaluation 

Computer Based Project 
City Schools-Prescott 
Syracuse, New York 13205. 

Gary D. Borich 
Institute for Child Study 
Indiana University 
Blooming ton, Indiana 

Harry L. Bowman 
Bureau of Educational Research 
College of Education 
Memphis State University 
Memphis, Tennessee 38111 



David J. Brown 

32 Havenwood Drive 

Brockport, New York 



14420 
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Leiroy P. Clinton 

R & D Center. for Handicapped Children 

Box 89 ' ; 

Teachers College 

Columbia University 

New York, New York 10027 



William H. Cormier 

Department of Educational Psychology 
CEB 108 

University of Tennessee 
Knoxville, Tennessee 37916 

< 

Gary H. deMik 
3340 Airport Road 
Ogden, Utah 84403 

Shirley c. Feldmann 

3333 B Henry Hudson Parkway 

Bronx, New York 10463 

Edward B. Fry 

233 Harrison Avenue 

Highland Park, New Jersey 08904 

Beatrice G. Green 
^7 East Street 
Stockbridge, Massachusetts 

Mark M. Greene 

Northwest Regional Educational 
Laboratory 

710^8 .W. Second Avenue 
Portland, Oregon 97204 

Franklin Greenough 
E 313 Health Sciences Bldg, 
University of Washington 
Seattle, Washington 98105 

Nairn C. Gupta 

Department of Jiducit tonal yuyi Unlny/y 
Ball State UnLveraJty 
Muncie, Indiana 47306 
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ERIC 



Donald L. Haefele 
203 Arps Hall 
College of Education 
Ohio State University 
Columbus, Ohio 43210 

Vynce A. Hines 
313A Norman Hall 
College of Education 
University of Florida 
Gainesville, Florida 32601 



Coy F. Hollis 
720 Moss Road 
M^phis, Tennessee 



38117 



Carl J. Huberty 
Fain Hall 

University of Georgia 30601 



Ernest B. Jaski • 
10438 S. Hamilton 
Chicago, Illinois 



60643 



John J, Kennedy • 

236 Ramseyer Hall 

The Ohio State University 

Columbus, Ohio 43210 

William E. Knabe 
W14 East Hall 

University Computer Center 
University of Iowa 
Iowa City, Iowa 52240 

William Landrum 
67 L^gare Street 
Charleston, South Carolina 



29401 



James C. Leavell r- 
44001 Garfield 

Macomb Intermediate School District 
Mouiit Clemens, Michigan 48043 

Robert J. Lesniak. 
Pennsylvania State -Uni vers ity 
The Capitol Campus 
Middle ton, Pennsylvania 17057 

*• " - ■ 

Luigi F. Liicaccini 
Dental Health Center ; 
14th. Avenue and Lake Street 



Walter M. Mathews 
1526 Simpson Street #3 
Madison, Wisconsin 53713 

Garry L. IfeDaniels 
1542 Live Oak Drive 
Silver Spring, Maryland 

Douglas McDonald 
P.O.- Box 142 

University, Mississippi 38677 



Lawrence McNally 
7 Monroe Heights 
Cortland, New York 



13045 



Sidney J, Micek 
Health Sciences Building SD-10 
University of Washington 
Seattle, Washington 98105 

Jerome Moss, Jr. 
125 Peik Hall 
University of Minnesota 
Minneapolis, Minnesota 55455 

John W. Newf ield 
355 39th. Street 
New Orleans, Louisiana 70124 



Herbert Ozur 
3407 Duke Street 
College Park, Maryland 



20740 



Roger P. Phelps 
718 Barn'^.s Avenue 
Baldwin, New York 



11510 



Laurel Anne Pickett 
16636 S.E, 26th. Street 
Bellevue, Washington 

Peter C. Rasmussen 
72^ Riga-Humford Road 
Churchville, New York 14428 

■•• ... • -.iJ^-- 
Cheryl L. Reed 

Educational Psychology SCC-G 
Purdue University 
Lafayette, Indiana 

Bruce G. Rogers 

6513 Pairhanlcc Chv^AAi- 



Dr. Louis Scheiner 
District Research Associate 
School District of Philadelphia 
21st and Parkway 
Philadelphia, Pennsylvania 

Irene Shigaki 

165 Christopher Street #5-J 

New York, New York 10014 

Bill C. F. Snider 

U14 East Hall 

University Computer Center 

University of Iowa 

Iowa City, Iowa 52240 

Tse-Kta Tcheng 
Computer Services 
Illinois State University 
Illinois 61761 

William H. Thomas • 
4436 Ti reman Avenue 
Detroit, Michigan 48204 

Orval Totdahl 
Rt. 3 Box 159 
Racine, Wisconsin 53403 

Robert Warmbrod 

208 Agricultural Administration Bldg. 
Ohio State University 
2120 Fyffe Road 
Columbus, Ohio 43210 

Mary Kathleen Zillioux 
93 Rock Beach Road 
Rochester, New York 14617 



The following biographical data were obtained from the 1971 presession 
application forms. Comparative data for the 1968 - 1970 presessions on non- 
parametric statistics are also reported, 
a. Sex of the participants 

Sex Frequency Percentage 





F 


1971 


1970 


1969 


1968 


Male 


40 


85 


90 


73 


71 


Female 


7 


15 


10 


27 


29 



As was true in previous years, the majority of the participants were males. 
b. Age of the participants 

Age Frequency Percentage 

20-29 
30-39 
A0-A9 

50 and over 

no response 

This year many of the participants (68%) were in the "under AO" category. 
There appear to be two major reasons for this large percentage of young parti- 
cipants: 1) professional staffs are acquiring younger msrabers who find they 
need an upgrading of their basic statistical skills;. 2) since many schools do 
not offer a course in nonparametric statistics, advanced graduate students and 
recent graduates find the presessigns'^an excellent opportunity for acquiring 
needed information/ 



F 


1971 


1970 


1969 


1968 


8 


17 


27 


10 


12 


24 


51 


23 


56 


47 


7 


15 


43 


27 


35 


5 


11 


7 


7 


6 


3 


6 









c» Institutional affiliation 

Type of Institution Frequency Percentage 





F 


1971 


1970 


1969 


1968 


College. or University 


.34 


72 


63 


54 


67 


Research Center or School District 


11 


24 


20 


23 


20 


Other 


2 ■ 


4 


17 


23 


13 



Most participants attended or held a position at a university or college; 
however f a nijmber of participants (24%) were employed by public school districts 
or federally funded research centers. 
d» Attendance at previous AERA presessions 



Previoxis Attendance 


Frequency i 




Percentage 




• 




1971 


1970 


1969 


1968 


Yes 


17 


36 


20 


38 


29 


No 


30 


64 


80 


62 


71 



Most participants (64%) were attending their first AERA presesslon. Of 
those who had attended previous AERA presessions, 4 recorded multiple presessions 
and 13 reported having attended a single AERA presesslon prior co this one. 
Past presesslon topics and numbers attending were: 

Multivariate Statistics 7 

Design and Analysis of Comparative 
Experiments '5 

' Systems Approach to Instruction ^ 
and Instructional Product De- 
velopment ; • 3 

Applied Linear Regression Analysis 2 

Computers and Natural Language 2 

Person-free & Item^^free Test Calibration 2 

Bayeslan Statistics 1 

Curriculum Evaluation 1 ' 



Inspection of this list suggests that most presesslon **repeaters'* had had 
previous presesslon experience in the area of statistics and research design. 
e> Years since doctorate 

Ye^is ' Frequency Percentage 









1971 


1970 


0 




16 


34 


37 


1-3 


• 

• 


18 


38 


37 


4-6 




6 


13 


13 


6-10 




2 


A 


7 


over 10 




5 


11 


7 



Most of the participants are either just completing their degree or are 
recent graduates in their first three years of completion. One of the 
purposes of the research training sessions has been to provide an opportunity 
for people who have been out of graduate school for a number of years to learn 
about new innovations in educational research. Obviously, this presesslon is 
not drawing extensively from the older audlance. 
f > Courses in statistics 

Number -Frequency 

1 .1 

- 2 .\ ■■ : ; 3- ; 

3 10 

4 13 

5 _ ; V 4 

6 or more 16 

As is evident from the table, the level of statistical competence as 
meaisured by the number^ of statistics courses was higher in 1970 and 1971 than 
for either of t^ie two previous years this presesslon was taught. 

The relatively low incidence of persons with minimal formal training in 
statistics (1 or 2 courses) reflects both self-selection on the part of the 



1971 


1970 


Percentage 
1969 


1968 


2 


00 


8 


10 


6 


, 17 


21 


22 


20 


10 


25 


29 


28 


20 


19 


16 


8 


23 


10 


4 


36 


30 


17 


19 



participants and some selectivity on the part of the co-directors in 
discouraging attendance of persons who did not appear to have the prerequisite 
entry skills (3 persons). 

Although we do not have comparative data from previous years to check our 
impressions against, we felt that this year's participants were decidedly 
different from past participants in their mathematical training and professional 
interests. The following course background in mathematics was reported: 



Number of courses in 
mathematics , exclusive 

of mathematics education Frequency Percentage 

. 0 14 30 

1-2 9 18 

3-5 . ■ 10 20 

6 or more 15 32 

When primary research interests of the participants were tabulated, the 

following distribution resulted: 

Primary research Interest . Frequency Percentage 

Experimental design lA 30 

Curriculum 11-23 

Measurement and evaluation 8 17 

Learning 3 6 

Reading 2 A 

Motivation ,2 A 

Creativity 2 A 

Administration 1 2 

Exceptional Children ~ 1 2 

Experimental psychology y X 2 

Attitude measurement 1 2 

Computers ' ^ 1 2 



Once again the heavy orientation of presesslon members in the areas of experi- 
mental design and measurement and evaluation suggests that for a substnntlal. 
minority of participants ;.( A 7%) the presesslon topic was one wliLch fell wiLhin 
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the domadLn of their major professional interest. Further evidence for this 
conclusion is found in a tabulation of number of courses taught in the areas 
of statistics I educational research and evaluation and measurement by presession 
participants. Of the 47 presession * participants , 27 of whom report teaching 
pne or more courses, 9 (33%) teach one or more courses in statistics, 17 (63%) 
teach one or more courses in educational research and 8 (30%). teach courses 
In evaluation and measurement. There are, of course, overlapping assignments 
in these three areas so that the percentages would exceed 100% were they added * 
across disciplines. 

Our Informal observations imply that for approximately 45% of our partici- 
pants, this presession was being used to meet the participants* need for 
advanced training in a specialty within their major academic discipline. The 
very active presession involvement of these participants, their Interest in 
teaching nonparametrlc statistics, and the high percentage of participants (67%) 
who expressed the intention of "maintaining some sort of contact with at least f 
one of the presession staff," suggest that this training session may serve as 
a stimulus for continuing study of and dissemination of nonparametrlc techniques 
in the Institutions to which the participants return. 

g> Allocation of duties among teaching^ research and graduate study (for those 
who responded to this item) 

. Teaching 

Frequency 



Amount of time ^ 



0-24% 
25-49% 
50-74%. 
75-100% 



Percentage 





1971 


1970 


1969 


1968 


14 


39 


50 


38 


13 




. 19 


19 


8 


24 


8 


23 


19 


' 38 


20 


7 


19 


13 


16 


43 
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Research 



ount of time 



Frequency 



Percentage 









1971 


1970 


1969 


1968 


0-24% 




11 


25 


7 


16 


13 


25-49% 




- 12 


27 


25 


9 


24 


50-74% 




9 


20 


29 


18 


20 


75-100% 




12 


27 


3'9 


57 


43 



ount of time 

0-24% 
25-49% 
50-74% 
75-100% 



Graduate Study 

Frequency 

3 
2 
4 
3 



Percentage 
1971 

25 
17 
33 
25 



The majority qf participants divide their time between teaching and research 
:tivities. Almost 50% of the people are spending at least half of their time 
I research activities but over 60% are teaching on at least a quarter time 
isis. This result is similar to that found in previous years. 

Articles accepted in refereed journals 



NuDiber 



Frequency 



Percentage 









1971 


1970 ^ 


1969 


1968 


0 




22 


47 


53 . 


53 


43 


1 




7 


15 


_ 3 


9 


12 


2 




4 


9 


17 


.17 


18 


3 




3 


6 


0 


9 


6 


4 




. 2 


4 


7 


6 


2 


5 ■ 




0 


0 


0 


2 


2 


6 or more • 




9 


19 


20 


4 


17 



■ • If. 



Number 


Frequency 




Percentage 








1971 


1970 


1969 


1968 


0-9 


33 


70 


53 


77 


78 


10 and over 


14 


30 


47 


23 


22 



It is Interesting to note that most of the participants actually have 

published very little to date. This is somewhat related to the fact that many 

are graduate students or have only recently finished their degrees. The profile 

of percentages conforms to the pattern set over the past three years. 

The number of research projects funded by USOE, NIMH, Qr other granting 
agencies . 



Number Frequency Percentage 







1971 


1970 


1969 


1968 


0 


22 


47 


. 60 


53 


59 


1-2 


15 


.32 


24 


33 


28 


3-5 


7 


15 


10 


10 


10 


6-10 


3 


6 


6 


4 


2 



The incidence of funded research projects for the presession participants 
Is slightly higher this year than in past years, but the differences are not 
sizable. Many participants have not had research projects funded by an outside 
agency. 

k> Geographic area of participants 

The presession participants came from 22 states with approximately equal 
proportions dtawn from the East and Midwest. 
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State 

California 

Florida 

Georgia 

Illinois 

Indiana 

Iowa 

Louisiana 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

New Jersey 

New York 

Ohio 

Oregon 

Pennsylvania 
South Carolina 
Tennessee 
Utah . . 
Washington 
Wisconsin 



1 
1 
1 
2 
3 
2 
1 
3 
1 
3 
1 
1 
1 

10 o 
3 
1 
2 
1 
3 
1 
3 
2 



Region 

West 

South 

South 

Midwest 

Midwest 

Midwest 

South 

South 

East 

Midwest 

Midwest 

South 

East 

East 

Midwest 

West 

East 

South 

South 

West 

West 

Midwest 



In the past, proximity to the location of the presesslon has been a factor 
ir attendance; however, attendance from the East has typically been less than 
predicted on the basis of proximity and attendance from the Midwest has been 
greater than would be predicted on the basis of proximity. Once again, this 
pattern is repeated for a presession held in the E.^st. 



Kegion 



East 
Midwest 
South 
Vest 

Canada & 
PuettoiRico 



Frequency 

14 
16 

- U 
6; ■ 



1971-East 

30 
34 
23 
13 



Pe^rcentages 
i970-Midwest 1969-West 1968-Mdwst 



17 
33 
23 
20 



31 
10' 
17 
29 

13 



25 
41 
14 
10 

10 



V, Proceedings 

A« Prior Scheduling 

The majority of the participants arrived in New York City the day pre- 
ceding the beginning of the presession. Information on hotel rates, the sched- 
ule of the lectures to be presented and an "advance organizer" for the preses- 
sion had been sent to each participant with his letter of acceptance. Arrange- 
ments for travel and lodging were the responsibility of each participant. 

B* Facilities 

The majority of the participants stayed. in the New Yorker Hotel during 
the presession. All presession meetings were conducted in a large conference 
room or In one of several smaller conference suites in the New Yorker. 
C« Presession Structure • ; 

Instruction covered a five day period with three lectures and one dis- 
cussion session each day. Midmorning and midafternoon breaks were used to en- 
courage participant-staff Interaction as well as to provide some, change from 
the lecture-discussion pattern of the ninety minute sessions. The prsession' 
participants and staff members were asked to wear specially prepared name tags 
during the* presession. 

A coiiq>lete set of lecture notes, multiple-choice-questions, discussion 
materials and a computer listing noriparametric statistical programs in a 
loose-leaf binder was given to each participant at the start of the^presession. 
Participants could refer to the multilithed lecture notes as the instructor . 
lectured. This allowed for a minimum of blackboard writing. Since we have 
always been plagued by a lack of board space, having the Lectures written up 
provided a valuable tool for effective" presentation of material. It was al<;o 
true that the lighting in the classroom left something to be desired so the 
printed lectures were even more appreciated. 
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/>The printed lectures were somewhat more thorough in coverage than the 
oral presentations since they Included derivations of the test statistics, 
and annotated references as well as an •^Intuitive** development of the tech- 
niques and examples of their use. The more extensive presentation provided 
by the written lecture notes was thought desirable since the participants 
would have permanent copies of the presesslon materials to be reviewed, at 
their leisure.' That the participants shared this view of the usefulness of 
the materials was indicated by the requests of several participants for ad- 
ditional copies of the presession materials. 



VI. Course Outline 

The planned schedule was presented to each participant prior to the start 
of instruction. The schedule was followed in its entirety and proved to be 
an ef fectiviB means of proceeding. 

TOPIC 



DAY 
Saturday 

January 30, 1971 



Sunday 
January 31 



TIME 



8:30 - 10:00 



10:30 
1:30 
3:30 
8:30 

10:30 
1:30 



12:00 
3:00 
4:30 

10:00 

12:00 
3:00 



Discussion of class objectives; ^ , 
Review of probabilistic concepts and 
of simpler ANOVA designs and post 
hoc analyses 

Wllcoxon Matched-pairs . test 

Friedman test and post hoc procedures 

Informal discussion . " ;* 

Marm-Whitney test (Wilcoxon and invt^rfilon 
forms) 

Kruskal-Wallis test and post hoc proccdurcH 

Application of the Wllcoxon test tO' blocked 
data; test for aligned observations 



3:30 - 4:30 



Informal discussion 



Monday 
February 1 



Tuesday 
February 2 



, Wednesday 
February 3 



8:30 ~ 10:00 
10:30 - 12:00 
1:30 - 3:00 

3:30 - 4:30 
8:30 - 10:00 

10:30 - 12:00 

1:30 - 3:00 
8:30 - 10:00 

10:30 - 12:00 
1:30 - 3:00 



Normal scores tests 

Irwin-Fisher Exact test 

Median Test, Chi-square test of 
homogeneity 

Informal discussion 

Confidence intervals for the 
tests for homogeneity 

Tests for equality of difference / 
across proportions 

Cochran's Q test 

Tests of independence and 
qualitative measures of association 

Rank measures of association 

Complete review of all material 
presented at the presession 



. The selection of particular non-parc\metric techniques for inclusion in 
this presession was made on the basis of the versatility of the techniques for 
a wide variety of research problems and their power relative to that of parar:etric 
procedures. Thus, the "quick and dirty" noniyarametric tests such as the sign 
test were ignored in favor of more powerful and more versatile techniques sueh 
as the Wilcoxon Matched-pairs test, the Hodges-Lehmann Alignment procedure, 
the Goodman tests for interaction across contingency tables. 
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VII. Evaluation 

Of the 47 participants who came to the presession, 46 were in at- 
tendance the first day, 46 the second day, 46 the third day, 39 the fourth 
day and 39 the final day. The drop in attendance from the third to the 
fourth day may be partially explained by a lack of heat in the building 
on the fourth day- 

At the conclusion of each day's lecture each participant was given 
a multiple choice test covering the day's material. The tests were turned 
in the/ following morning and corrected. A total of four tests were given 
and corrected. . No special effort was made to collect exams from reluctant 
participants, since this might have created an uneasy situation for the 
rest of the session. Copies of the exams are presented as an appendix. 
The statistics for the four tests are as follows: 



Test 


// Items 


N 


Mean 


S.D. 


1 


13 


33 


9.06 


1.94 


2 


12 


27 


9.96 


2.33 


3 


13 


25 


9.64 


2.80 


4 


12 


23 


9.83 


2.72 



The staff was pleased with the performance of the participants on 

V 

the exams. Th6' majority of the participants made a concerted effort to 
attempt the questions and thus consolidate the learning that had taken 
place during the day. 

The "Evaluation By Participants" questionnaire supplied by AERA was 
given to participants at the end of the two morning sessions on the final 
day of our presession. Thirty-nine of the 47 participants or 81 percent 



completed the questionnaire. The number responding to each option of an 
item and the corresponding percentage of total responses to that item are 
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/€RA 1971 ^SEARCH Training Sessions 
Evaluation by Participants 

Name of .^p^^f^mN! non-parametric methods 
Director: 

IA. To what extent did the relative availabil''*/ or unavailability of books 
and journals i nterfere wi th or promote your attempts to master The con- 

I tent of this session? 

I 

NON E 3(8 %) LITTLE 13 (33%) SOME 23 (59%) 

IB. To what extent did reproduced materials given to you by the staff improve 
matters? 

HELPFU L ?. (5%) VERY HELPFU L 37 (95%) 

2A. Did you feel that you lacked a "place to work", either alone or in small 
groups? 

YES 3 (8% ) N O 33 (85%) No COMMEN T 3 (8%) 

2B. Was your room sat i 9. f actory? 

YES 19 (49%) N O lA (36%) No COMMENT 6 (15%) 

3A. Which features of the meeting rooms were inadequate or not conducive to 
learning? / 

BLACKBOARDS 24 (62%) LACK OF OVERHEAD 

PROJECTOR ^ (105^) 



SIZE 4 (10%) 

FURNISHINGS 1 (3%) 



LIGHT 


11 


(28%) . 


AIR 


13 


(33%) 



MISCELLANEOUS. 15 (38%) 



38. Which features were especi a I I y„f aci I i tati ve in the same regard? 

LIGHT 1 (3%) AI R 0 (0%) 

SOUND 6 (15%) FURiJiSHINGS 8 (21%) 



CONVENIENCE 15 (38%) Mo COMMENT I'J O'^^-) 



B9 ^151 
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4A. Wiis five days too long a period to leave your work £t home for The D_r:c3: 
• of attending this session? 

YES 8 (21%) N O 30 (79%) 

4B. Was five days too short a period in which to learn much of the content c- 
this session? ^ . • 

YES 15 (39%) NO 23(61%) 



5A. Were you allowed enough time in which to pursue activities of your own 
choosing? 

YES 21 (55%) N O 8 (21%) No COMMEN T 9 (24%) 
5B. Would you have preferred not to meet in the evening after dinner? 

YES , IP (26%) N O 8 (21%) No COf^MENT 20 (53%) 

5C. Would you have preferred more or fewer meetings per day than there actual iy 
were or was the number of meetings per day agreeable to you? 

FEWER 3 (8%) ENOUGH 32 (84%) MORE 3 (8%) 



6A. Were the Individual lectures too long to sit and listen or take notes? 
YES 7 (18%) N O 31 (82%) 

6B. Were the lectures scheduled inan appropriate sequence? 

YES 32 (86%) NO 5 (1A%) 



7. Did you have sufficient opportunities to interact with other participanTs? 
YES 30 (79%) N O 8 (21%) 

8A. Were the instructors too inaccessible or unapproachable so that you did rot 
get the individual attention that you desired? 

YES 0 (0%^ N O 35 (92%) No COMMENT 3 (8%) 

8B. Was It helpful to have graduate student assistants present? 

■ /-*' — ■ 

YES 17 (46%) N O 5 (14%) No COMMENT 15 (40%) 
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9A, Did the content of the lectures and readings presuppose more previous 
training than you had? 

YES. 11 (29%) NO 27 (71%) 



9B. Did tho content of the lectures and readings presuppose iess previous 
training than you had? 

YES 2 (5%) NO 36 (95%) 



10. To what extent was the content of the lectures and readings relevant to 
what you hoped to accomplish during the session?. 

SOME 5 (13%) MUCH 3A (87%) No COMMENT 0 (0%) 



I lA. Were the lectures stimulating and interesting? 

YES(usual ly, or somewhat) 22 (56%) VERY 17 (44%) 

MB. Were the lecturers competent to speak on the subject assigned them? 

YES 39 (100%) N O 0 (0%) 

lie. Were the lecturers well prepared? 

YES 39 (100%) NO 0 (0%) 



12. Were you disappointed in any way with the group of participants? 

YES 0 (0^^) ^ N O 39 (100%) 

13. If you had it to do over again would you apply for this session which you 
have just completed? 

YES 39 (100%) NO 0 (0%) 



14. If a session such as this is held again would you recommend to others li krj 
• you that they attend? 

YES 39 (100%) N O 0 (0%) 



15. Do you anticipate maintaining some sort of contact with at least one of the 
session staff? 

YES 26 (67%) NO 1 3 (33%) 
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16. Do you feel that AERA is making an important contribution to education 
by sponsoring sessions such as this one? ^ 



17. Do you feel that anything has happened during these five days to make it 
more likely that you will leave your present position of employment? 

YES 4 (11%) NO 33 (89%) 



18. Is it likely that you will collaborate in research with someone else 
attending this session . (other than those you already were likely to 
col laborate w! th)? 

. YES 9 (24%) NO 29 (76%) 



19. Do you think that the staff should feel that it has accomplished its ob- 
jectives during this five-day session? 

YES 38 (100%) N O 0 (0%) 
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Because of the wording of item lA, the responses are difficult* 
to interpret. Either Jthe participants felt that the limited number of 
books and articles that we supplied were of some help, or that our 
supply of books and articles was so lluited as to have interfered with 
their attempts to master the content of the presession. Responses to 
item IB indicate that our 327 pages of reproduced materials were help- 
ful. Items 2A and B and 3A and B suggest that the participants were 
generally satisfied with the physical arrangements, singling out con-; 
venience as a strong point and blackboards, lack of heat for one day 
and light as the weak points. Responses to items 4A and B suggest that 
if anything 5 days is perhaps too short a period of time for several of 
the participants. Responses to items 5A through 7 indicate that par- 
ticipants approved of the schedule of lectures and discussion sessions . 
used in the presession. Responses to questions 8A and llA through C 
indicate that the instructors were perceived as being accessible and 
competent and their lectures were well prepared and generally interesting 
and stimulating. Responses to questions 9A through 10 indicate that the 
content of the lectures was highly relevant to the expectations of the 
participants and in general was presented at an appropriate level of 
difficulty. Responses to questions 12 through 16 and question 19 
clearly indicate that participants had a high degree of^ satisfaction 
with their participation in our presession. 
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Directors ' Evaluation 

The staff felt that our presession was highly successful in nearly 
all respects. The staff's perceptions were overwhelming supported by 
the participants' responses to the evaluation questionnaire. An im- 
portant contributor to the success of this year's presession is the 
experience gained from 3 prior offerings of similar presessions. The 
staff were all familiar with each other and, with the exception of the 
assistant, were used to working together. The unanticipated loss of 
one staff member due to illness forced a last-minute reassignment of 
staff duties, but the presession instruction did not seem impaired by 
a few somewhat impromptu presentations. The staff also felt that the 
backg^found of participants this year was slightly better than in pre- 
vious years and that this also contributed to the success of the pre- 
session. 

The presession started on schedule with almost no time lost to 
the administrative details of getting under way. The presession stayed 
on schedule through out and all the material that we had intended to 
cover in the f ive days was covered. In addition, there was" reasonable 
time for the spontaneous discussion of topics that were raised by the 
participants during the more formal presentations. 

A friendly and somewhat informal relationship developed between 
staff and participants both during lectures and discussion periods. 
The participants indicated informally and on the evaluation question- 
naire that the content of the presession was consistSfent with their 
expectations and not out of line in terms of difficulty level. A51 
in our past presession, the materials provided to each part Lc I pant 
at the beginning of the presession were seen as extremely valuabJe. 
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The most unfortunate aspect of the presfession was the lack of heat 
in the hotel on the fourth day. It became cold enough that participants 
wore outside coats and gloves. Milder contributions to this lack of 
comfort were that the main lecture room was a little too small and had 
inadequate blackboards and lighting. 

' The initial uncertainty about funding and the subsequent con- 
ditioning of whether, a presession would be offered or not on, the basis 
of having at least 20 paid applicants was unfortunate. As a result the 
several months preceding February were rather trying on the codirectors. 
There was some initial hesitancy to prepare materials for a presession 
that might not be offered. When it became clear that our presession 
would be given, there was quite a bit of work to be done in a short 
period of time. We hope that future directors of presessions will not 
have to labor under similar conditions of uncertainty. 

For future presessions we recommend that applications received 
be allowed to accumulate without action being taken on them until most 
applications have been received. Obviously, action would still have to 
be taken sufficiently in advance of the presession to allow for the 
accepted applicants to make the necessary arrangements for their at- 
tendance. If the pool of applications was larger than the number that 
could be accepted, then appropriate criteria could be used in selecting 
the "best" applicants^ This year we started by accepting all but the 
most obviously unqualified applicants until we reached a number slightly 
•» 'larger than what we had initially set as an upper bound. The results 
were that several of the later applications were even more desirable 
than some already accepted but were rejected because no more room was 
available. . 



Appendices 3 
Tests administered to participants 
in Non-parmetric Statistics 



NAME: 

TEST I 

Material Covered 

a) Introduction to nonparametric statistics 

b) The Wilcoxon Test for Matched-Pairs 

c) The Friedman Test 

To be returned at 8:30 A. M. on January 31 

I. Nonparametric tests are also called distribution-free tests, but 
distributions such as the chi-square distribution are used in de- 
termining whether the test statistics are significant. To what 
does the term "distribution- free" apply? 

a) the population (of observations) 

b) the sample (of observations) 

c) the sampling distribution of the test statistic 

d) all of the above 

e) none of the above 

2. The term "conservative test" when used in reference to a particular 
nonpsirametric test means that: 

a) the test makes fewer assumptions about the population than 
does its parametric analogue 

b) the probability of a type I error for the test is less than or 
equal to the upper bound, o , 

c) the test does not permit inferences to be drawn about the 
population 

d) the nonparametric test is more powerful than its parametric 
analogue when the assumptions for the V^tter have been violated. 
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3. To interpret a siaiemi'ni thai ilu* a&ymptotic rciativr iMIu ioncy 
of the sign test is . 637, we must know: 

a) the test statistic with ^Vhich it is being compared 

b) the distrioution of the parent population 

c) both of the above 

d) none ot the aoove 

4. The Wiicoxon matche'l -pairs test is the nonpar ametric analogue of the 

a) pedred t test 

b) two independent-sample t test 

c) one-way ANOVA F test 

d) none of the above 

5. The procedure . of discarding pair differences equal to zero when 
computing the WilcoAon matcaed pairs test statistic is 

a) desirable because it discards oDservations which give no 
information 

b) desirable because it permits the use of exact tables 

c) undesirable because it sjpuriously increases the power of the test 

d) undesirable because it reduces the power of ihe test 

6. When parametric assumptions are met^ the Wilcoxon matched-pairs 
test is more powerful than the 

a) Sign test 

b) Fitiher randomization test for matched pairs 

c) Corresponding ANOVA F test 

d) All of the above 

Questions 7-9. An investigator is interested in teachers' use of various 
types of questions in teaching mathematics. He identifies 4 types of questions 
whicn demand responses of different levels oi complexity. He records the 
number of questions of each type asked by each teacher in a random sampie of 
10 teachers. The frequencies are reported for teacher and question type. 
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Question 
type 



•achor 
A 
B 
C 
D 
E 
F 
G 
H 
I 
J 
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'» 


4 


n 


1 
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• f . 
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7 
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7 


2 


5 


I 



7. The most appropriate nonparametric test for these data would be: 

a) Fisher randomization teat 

b) Friedman test 

c) Wiicoxon matched -pairs test 

d) Other specify . • 

8. The null distribution of the nonparametric test would be asymptotically 

a) standard normal 

b) <?hi-square with 3 degrees of ireedom 

c) chi-square with 9 degrees of ireedom 

d) chi-square with 10 degrees of freedom 

e) cni-square with 27 degree? ol freedom 

9. Alter rejection of the null hypothesis. the''investlBator could set up a 
contrast to determine whether the fourth type of question difiered from 
the other three types. The procedure(s) used to determine whether 
this contrast was significantly different from zero would be: 

a) Large -sample procedure using ^ 

b) Rosenthal and Ferguson procedure using / constant - F 

c) Both of the aoove 

d) None oi the above 



10. To c uU uImIc tlu' l''fio<lm:ui tosl <>m> iinisl 

a) rank tlio obsorvatior.s iis oacli block sepJU'aU'ly :unl I in* I Ih. 
sum of ranks for each block 

b) rank the observations in each block separately and find the 
sum of ranKs for each treatment 

c) rank all of the observations disregarding blocKs and find the 
sum of ranks for each block ti 

d) rank all of ihe observations disregarding blocks and find the 
sum of ranks for each treatment 

11. The parametric analogue of the Friedman test is the 

a) one-way ANOVA F test , j 

b) two-way ANOVA F test 

c) Wilcoxon matched pairs test 

d) two independent samples t test 

12. The post hoc procedures for significant Friedman test define contrasts 
to be tested in terms ol the 

a) treatment population means 

b) treatment population medians 

c) treatment expected average ranks 

d) treatment population variances 

13. An administrator who is interested in the possible effect of school 
district consolidation on pupil-teacher ratios compares the median 
pupil-teacher ratios for 60 presently consolidated districts before 
and after consolidation. He believes that the underlying distribution 
of pre-posi consolidation differences is extremely positively skewed. 
Indicate the most and least appropriate test statistics respectively 
for his hypothesis . 

a) Wilcoxon matched -pairs teit. Fisher randomization test 

b) Sign test, Wilcoxon matched - pairs test 

c) Fisher randomization test, Sigh test 

d) Wilcoxon matched -pairs test. Sign test 
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TEST 2 . 

Material Covered 

a) Mann-Whitney Test 

b) Kruskal-Wallis Test 

c) Wilcoxon Test on Block Designs and Hodges -Lehmann Test 
on Aligned ObsprvatioiiK 

To be returned at 8:30 A. M. on February 1. 

1. In performing an experiment when comparing two samples, let one 
sample contain 2 subjects and the other 5 subjects. Assume ranks 
from 1 to 7 are to be assigned to the scores of the two samples. 
How many combinations of ranks can the sample with 5 subjects take on? 

a) 7 ■'• 

b) . 10 

c) lib 

d) 42 

v) none of the above. 

2. Let E(Tj) = (nj,./2) (n^ + n^ + 1) whpre T^ is the Wilcoxon statistic. 
If the Mann-Whitney statistic U equals T^ - (n^/2) (n^ + i) where 
n^ = 4 and n = 9, find the E(U). 

a) 10 

b) 18 

c) 20 

d) 36 • . • 

e) 45 

Questions 3-5. An investigator has randomly selected 48 students and 
randomly assigned 12 students each to one of 4 experimental conditions. 
At the conclusion of his experiment he finds that for each group the test 
scores have a very pronounced negative skew. He wishes to determine 
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thrT the four distributions diner in location. 



'i. The? most appropriate nonpar am ctric test for these data is: 

a) the Friedman test 

b) the Hodges -Lehmann test for aligned observations 

c) . the Kruskai-Wallis test . 

d) the Mann -Whitney (WiicoAon) test 

4. ir tho investigator in assigned midranks to all tied scores 
but made no other corrections for ties in his test statistic, the 
test statistic in this form would be numerically 

a) larger than the test statistic fully corrected for ties 

b) smaller than the test statistic fully corrected for ties 

c) exactly the same as the test statistic fully corrected lor ties 

5. After rejection of the hypothesis pf equality of performance under 
the various experimental conditions, the investigator could set up 
contrasts to determine whether there were any pair-wise differences 
among the treatment groups. The procedure(s) used to determine 
whether these contrasts were significantly different from zero would br: 

a) Large-sample procedure using Vx^ 

b) Tobach, Smith, Rose and Richter procedure using exact tables 

c) Rosenthal and Ferguson procedure using ^ constant ^ F 

d) Both (a) and (b) 
o) Both (a) and (c) 

6. The major reason for selecting a nonparametric test for (3) would probably 
be suspected violation of the assumption of: 

a) continuity in the parent populations 

b) equality bl variance in the .parent populations 

c) normality in the parent populations 

d) equality of the pair-wise correlations of treatments in the 
populations 



Questions 7 - «. Sief/cl and Tukey tJovclopetJ a nonparametric test to compare 

tho relative; sprerad (dispersion) ol two distributions. To perforri the U.-st, 

write the observations ironn samples A anO B in increasing order, but assign 

the LOW RANKS to the MOST EXTREME OBSERVATIONS and the HIGH RANKS 

to the LEAST EXTREME OBSERVAirNS. For example, 

^i orv 0 :i 5 b 8 8 10 10 11 12 13 13 l4 15 16 17 19 25 

Sample ABABAABBBBRBAABAAA 
Hank 1 4 5 8 9 12 Vild 17 1« 15 14 1 1 10 7 6 3 2 

The test statistic is the sum of the ranks for sample A. 

7. The nonparametric test to which the Siegel-Tukey test bears the 
greatest resemblance is the: 

a) Friedman test 

b) Hodges-Lehmann test of aligned observations 

c) Mann-Whitney test 

d) Wilcoxon Matched -Pairs test 

H. If the Siegel-Tukey test is used to test the hypothesis that populations A 
and R are the same in dispersion against the alternative that the 
dispersion in population A is greater than that in population B, the 
null hypothesis will be rejected if the sum of the ranks for sample A is 

a) "too small" 

b) "too large" 

c) . either "too large" or "too small" 

9, The Hodges-Lehmann Method of Alignnient in the two sample case is 
used to test equality of two distributions with respect to: 

a) proportions (i. e. , p = p ) 

b) location 

c) dispersion (i. e. , aj « a? ) 

d) correlation between variables X and Y 
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Test 2 Page 4 

The large-sample statistic for the Hodges -Lehrhann Method of 
Aligned ooservations has a sampling distribution that is approxinia1c>ly 

; when each block consists of a t reatment and ronlrol 

group. 

a) ^^^^ " 1^ " 1) degrees of freedom 

b) N(0,1) 

c) t with + - 2 degrees of freedom 

d) F with and (R-1) (C-1) degrees of freedom 

e) none of the above 

The Hodges -Lehmann Test on Aligned Observations uses: 

a) intra block differences in the observations alone 

b) inter block differences in the observations alone 

c) both intrablock and interb lock differences in the observations 

d) neither intra block nor interb lock differences in the observations 

To test vvhether training can increase the amount of a certain chemical 
in the brain, three litters of rats were each divided at random into a 
control group and a treatment group (which received the training). 
The amounts (in appropriate units) of the chemical present after a 
given period of time are shown in the following table. 

Litter . . Control Treated 

.1 3 . .* . : 1.3 2.9 

2 B.2 4.0 3.5 4-5 

3 . 9.5 10 . 0 10. 5 

The most appropriate test to use in testing the hypothesis that training 
has np effect on the amount of chemical present in the brain is the: 

a) Friedman Test 

b) Hodges -Lehmann Test of Aligned Observations 

c) Wilcoxon Matched -Pairs Teat 

d) Kruskal-Waliis Test 

e) Mann-Whitney Test 
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TEST 3 

Material Covonxl ^ 

a) Normal Scores Test 

b) Irwin- Fisher TOxacl Test 

c) Median Test and Chi- square test of Homogeneity 
To be returned at 8:30 A.M. on February 2 

1. The Chi-square test of homogeneity for a 2 x 2 table is the large 
sample form of 

a) the Fisher Exact test 

b) the two independent - sample t-test 

c) the Z-test where Z = - 

d) none of the above 

2. The Fisher Exact test for a 2 x 2 table assumes: 

a) independent random samples from identical continuous distributions 

b) matched-pairs o( observations from a binomial distribution 

c) independent random samples from binomial distributions 

d) matched-pairs of observations from a hypergeometric distribution 

3. When the null hypothesis for the Fisher Exact test is true, the resulting 
test statistic has an exact 

a) t distribution 

b) binomial distribution 

c) Chi-square distribution 

d) normal distribution 

e) hypergeometric distribution 
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Test 3 l'a«u 2 

4. Which of the following Is NOT an assumption needed to insure valid 
use of the chi-squaro approxin^ation to the Fisher Exact Test? 

a) V- the expected t:c;ii frequencies should bo greater than or equal 

to five 

b) observations are independent 

c) underlying distribution ot the variable of interest is normal 

d) probability of the qualitative Variable is constant over subjects 
c) all of the above assumptions are needed 

Questions 5 - G. In a study of reading ability and syntactical mediation in 
paired -associate learning an investigator reports: 

. . the fourth grade distribution of mediators and non-mediators 
approached a significant level, suggesting that good readers may be 
more likely to mediate spontaneously. " 

His data are: 

Table 1 - Frequency of Spontaneous Use of Syntactical Mediation 

Mediators Nonmediators Total 

Good Readers 5 » 

Poor Readers 1 » 

I 8 



What nonparametric test would you recommend? 

a) Chi-square test of homogeneity 

b) Fisher Exact test 

c) Median test 
Normal curve test for equal proportions 



d) 



What statistical hypothesis would you test for the data in Question (5) ? 
a) pj » P2 

W Pr(Xi > Mo) - Pr(X2 > Mq) 

c) Pr (Mediator/Good Reader) = Pr (Mediator/Poor Reader) 

d) of - 



116 



Test 3 Page 3 

7. Which is not a condition needed to insure valid use of the Chi-square 
approximation to the Irwin-Fisher test? 

a) the expected frequencies should be 5 or greater 

b) independence among the oDservations 

c) the underlying distribution of the variable is known 

d) the probability of the qualitative variable is constant over subject 

8. In the computation of the median test, each sample is dichotomized 

a) at Its own median 

b) at its respective population median 

c) at the pooled median 

d) at the combined population median 

e) in some inexplicable fashion 

Questions 9 - 10: An investigator interested in the stability of self- ascribed 
social Class constructs the following table of data based on two independent 
survey samples (a 1945 and a 1963 survey): 

Table of frequencies 

Self- ascribed 

social class 1945 survey 1963 survey 

Upper and xipper 33 89 

middle 

Middle and lower 472 463 

middle 

Working and lower 570 I68 

9. ^ Which of the following tests should the investigator use? 

a) chi-square test of homogeneity 

b) Fisher Exact test 

c) normal curve test for equal proportions 

d) other, specify^ 
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10. Tho null distribulion ol \Uv \v^\ slalisUr i-lu)son in (!)) is 

a) standard normal 

b) chi-square with 2 degrees of freedom 

c) chi-square with 3 degrees ol freedom 

d) chi-square with 5 degrees of freedom 

e) other, specify 

11. When the median test is used to test the hypothesis, H^i u = u 
the following must be assumed: 

a) symmetry of the populations 

b) normality of the populations 

c) both of the above 

d) none . of the above 

12. The major advantage clt^d for the use of normal scores tests is: 

a) computational ease and s^peed 

b) availability ol extensive, exact tables for the K-sample problem 

c) high asymptotic efficiency relative to their parametric analogues 
even when the assumptions for the latter have been satisfied. 



d) other. 

specify- 



13. Large-sample normal scores tests in the two- sample problem are 

referred to the distribution to determine statistical significance. 

a) standard normal 

b) t with n^ + ng - 2 degrees cf freedom . 

c) F with 1, and n^ + - 2 degrees of freedom 

d) Chi-square with n^ + n^ - 1 degrees of freedom 
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NAME: 



Mnlrrial Covci'i'd 

a) Confidence Interval rroccdureK lor the Chi-square antMli<« 
Median Tests. 

b) Test for Equality of Differences across Proportions 

c) Cochran Q Test 

To be returned at 8:30 A. M. on February 3 

1. An investigator who uses a Chi-square test for homogeneity to test 
"o- ^1 " ^2 " • • • = P7 in a 2 by 7 table rejects his hypothesis! He 
decides to use the Chi-square analog to Schefle's theorem to examine 
differences among the P^^. The value of ^ to be used in the 
confidence interval would have how many degrees of freedom? 

a) 1 

b) 2 

c) 6 

d) 7 

e) indeterminate from the information given 

2. How many confidence intervals may the investigator of question 1 
set up if he uses the Chi-square analog and wants to hold the overall 
probability of a Type I error to . 05 ? 

a) 1 

br 6 

c) 7 

d) 21 

e) an unlimited number. 
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i. Which of the following is not a valid contrast? 

a) - 

b) Pi ^ ^2 ^ ■ "^^4 



c) Pj - P2 - 








d) 2p^ + p2 ■ 
















The variance of the contrast p - 


p is estimated by: 


a) P3q3 ^ 


P4^4 


d) 


P3^3 - P4^4 


"3 


"4 




"3 "4 




P4^4 




none of the above 


"3 ' 








^> - 


P4^4 






"3 


"4 . 







5. The principal reason for using the arc sm transformation, 

=2 arcsin , in place of p, in test statistics and 

confidence intervals is to: 

a) test the hypothesis of equality of the populations in terms 
of more easily interpreted parameters 

b) simpiiiy computations 

c) increase the degrees of ireedom in the computation of the 
test statistic and confidence intervals 

d) . guarantee that the null distribution of the test statistic is 

approximately Chi-square 

e) increase the power of the test 
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Tho tf'sl ol 11^^; Ai « « ... = Aj^ = Ao where = Pj|^ ^ p^,^ 

is the K -sample? extension of the . lor the? cjisc ol 

Uirg^ samples, 
n) The l-tost 

h) Tho Irwin-Fisher Kxad lest 
k ) Tne Wilcoxon lest 
d) The Friedman test 
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K 

7. F'or valid use of the test U' = E W, (A, - A^)' 

o pq=i k ^ k 0' 

a) The underlying variables must have equal standard deviations 

b) The sample sizes must be equal 

c) The underlying variables must be correlated 

d) The values of must be equal for all k (k = 1, 2, . . . , K) 

IK 

and the values of p^^^ must be equal for all k (k = 1, 2, . . . , K). 

e) The sample sizes must be large 

8. Cocliran's Q iray be used to test data which could also be analyzed 
by means of 

a) a k-sample Chi-square test of homogeneity 

b) a one-way analysis of variance 

c) a randomized block analysis of variance 

d) an n X k contingency table with both margins free. 

9. For the following table, the computed value for 

Q - K(K-1)I(T J - ITj) 

TREATMENT 

I n III 

Subject 111 0 

a) 1/2 21 n I 

b) I. a 1 I I 

c) • 5 ' 4 U 0 1 

d) 1-1/.'. .:-.(!! I 
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Questions 10 - 12. A researcher is interested in deternnining whether children's 
prolVrcMicos for form versus color change as the children grow older. Sl)c 
rolUu'ls longitudinal data for a random sanipU» of 25 cliildron who arc ti^stCd 
at agrs 2 l/2, 4, ^ 1/2, and T). At c^acli testing, each child is 

clas.sified as fornn -dominant (score = 1) or color dominant (score = 0). Tlic 
null hypothesis states that the probability of a form-dominant classification 
does not change over time (i.e., the proportion of form-dominant classifications 
remains constant. ) 

most appropriate nonparametric test for this hypothesis would be: 
chi"Square test of homogeneity 
Friedman test 
Cochran Q test 
Kruskal-Wallis test 

11. The null distribution ot the nonparametric test statistic would be 
asymptotically: 

a) standard normal 

b) chi-square with 5 degrees of freedom 

c) chi-square with 6 degrees of freedom 

d) chi-square with 24 degreesvof freedom 

e) chi-square with 120 degrees of freedom 

12. If the null hypothesis of equal proportions of form-dominant classifications 
over time is rejected, can the investigator use a contrast with orthogonal 
polynomials to determine whether there is a linear trend in the proportion 
of form -dominant classifications? 

a) Yes, such a contrast is justified. 

b) No, such a contrast is not justified because the proportions are 
correlated 

c) No, such a contrast is not justified because the levels r>r lln 
independent variable do not satisfy the condition.s for Drtiifj^onnl 
polynomials 
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10. The 
a) 
b) 
c) 
d) 



NAME: 



, TEST 5 

Mai(»rial C'ovi*ri*d 

a) , Tests of Independence and QuXlilatiyc Measures of Association 

b) Rank Measures of Association 



1. For a Chi-square test of the independence of two categorical variables 
it is necessary that 

a) the underlying distribution of both categorical variables be normal 

b) the frequencies in the cells result from categorization of 
independent units in the sample 

c) the frequencies be normally distributed 

d) the marginal frequendes be. known a priori 

2. The square root of the mean square contingency coefficient and the 
Pearson Product Moment coefficient are equivalent for contingency 
tables where 

a) R and C both equal 2 

b) R and C are equal 

c) R equals one and C can take on any value 

d) C equals one and R can take on any value 

Questions 3-4. An investigator wants to determine the reliability of 12 readers 
who are charged with rating the performance of doctoral candidates on their 
comprehensive exams. To assess the reliability of the raters, all 12 are asked 
to rank order the same six candidates. 

3. If the investigator wants a single reliability coefficient falling between 
0 and 1 inclusive, what would you recommend? 

a) Cochran Q test 
Friedman test _ 

c) Kendall Coefficient of Concordance 

d) Spearman Rank Order Correlation Coefficient 

e) Coefficient 123 ' . 



Vy^.i^j^>r8:&t^a«^<l;v^r^«T>r^.**lO^«>^«^»^«*■w/^»^e?xvw 
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4. Thv investigator decides that he wants the added assurance that 

his nu^nsiuv or relnihilily will l>i' sini isiii-ally sip[i)iru-an1 (sImI islu :Miv 
(linVmil from 0) ;i1 »^ .1)1. llu* :ipp^*<*P»'i*'h» lMhl(\s lo ih h riniiir 

Imiw hirjii* llu' I'oiH'rieiiMii musj lu* \\\ ()r<l(»r lo muhM this i^rilcr i!>n. 

a) at least .10 

b) at least .25 

c) at least .37 

d) at least . 50 

e) none of the above 

5. The number of degrees of freedom for a chi-square test of independence 
in a 3 X 4 table is: 

a) ^ 4 

b) 6 

c) n 
\.d) 12 

c) none of the above 

6. The parametric analogue to thq chi-square test of independence is: 

a) one sample test for correlation coefficient equal to zero 

b) two sample t-test for independent samples (means) 

c) two sample t-test for related samples (means) 

d) F-test for columns in twb way ANOVA fixed effects model 

e) none of the above 



The Spearman coefficient of correlation is identical to a Pearson Product 
Moment correlation coefficient if the two variables being correlated are 

a) both dichotomous 

b) one dichotomous and one continuous 

c) both rank order values 

d) both continuous 



ERIC 
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. Test 5 Page 3 

tt. Two advantages of Kendall's lau over the Spearmd!h rank order correlation 
cM)c»Hi('ion1 nro 

a) ^ is normally distribiilril Tor n * 10 and ran In* rxlrndrd \o 
partial correlations 

b) T is easier to compute than r^ and tends to be somewhat more 
conservative 

c) is normally distributed for n ^ 10 and is esisier to compute. 

d) T has no advantages since it is algebraically equivalent to r . 

s 

9. The value oi Keiidallfe ^ for the two rankings ^' 
Ranking A: 1 2 3 4 5 6 7 

Ranking B: 2 1 3 4 7 5 6 
is 

a) 17/21 

b) 1 3/7 
. c) 11/21 

d) 5/7 

10. A measure of the degree of agreement among several judges' ranking of 
a set of objects is the 

a) Spearman correlation coefficient 

b) Phi correlation coefficient 

c) ^. Mean square contingency coefficient 

d) Kendall coefficient of concordance 

11. The mean square contingency coefficient denoted by has . .. 

a range of possible values from 

a) -I to +1 

b) 0 to +1 . 

c) -min (R-1, C-1) to +min (R-1, C-l) 

d) 0 to min (R-1, C-l) 

e) none of the above 
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12. Tlu' luimhiM' <>r ili'firiu's ol rriuMfiun lor llu* (Mn-S(iiiaiM' ti'Sl of iiutciM iuliMu t' 

slntislic wluMi ralculalocl on iu\ IM).y C I'Dntin^riu w is " 

a) n - 1 
I)) n-R-C^-l 

c) RC 

d) (R-1) (C-1) 

13. The numerical value of the measure of association (|)' (0 £ ({>' £ 1) lor a 
3x4 table for which N = 100 and thp computed chi-square is 19, 22 is 
(to the nearest hundredth): 

a) .10 

b) .18 

c) -25 

d) . 31 

e) .44 

' ■ ' \ ' . 

14. In a study in which doctoral students in Counseling and in Higher Education 
were sampled, the two groups of students were asked whether they 
favored or opposed foreign language exams and whether they had or 

had not completed their language requirements. It was found that 

« •0283 and Y„t,«3.216 and Ub' -12, 26 was significant at -05. 

A valid statistical conclusion to bq drawn on the basis of these data is: 

a) for doctoral students in C, favoring" language exarns and having 
completed the language requirements are unrelated. 

b) for doctoral students in HE, favoring language exams and having 
completed the language requirements are positively related. 

c) The confidence interval for - will lie completely 
below zero 

d) all three conclusions can be made 

e) none of the three conclusions can be made 
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Test 5 Pa^e 3 

15. Tho test ol H : Yi = ... =jr„ against H, : II is 

false, where = log^\ .^^C^lVk "** ^°8eP2Zk 1083^12^ " ^pSeP'^lk 
assumes: 

a) K match paired samples from binomial distributions, 

b) 2K independent random samples from binomial distributions. 

c) 2K correlated samples from hypergeometric distributions. 

^ d) K large independent random samples from K bivariat^ 

t . ' . ■ . . . ■ . ■ 

fi qualitative variable universes. ^ 
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Z. Gr.NERAL DF,SCmPT10N: 

This profession will exposJ educational p iD^ini no personnel and school 
administrators to the use and application of management science and operations 
research models to problems In education.' Sessions wi I I be devoted not only 
to theory and techniques necessary for Improved resource allocation to and 
planning of school act i vitles, but also to explanations of a number of recent 
successful applications in education. Throughout the session, the utility 
and limitations of such models to Improve and effect changes in educational 
planning and administrative practices will be stressed. 
Applications for this session are invited from educational planners and 
administrators, from the local, state, and federal levels (of education) 
Involved In the allocation and administering of resources (financial, 
facilities, and personnel) to elementary and secondary school activities. 
Participants then would Include USOE staff, regional laboratory personnel, 
State Education Department staff, professors of educational administration, 
local school administrators, and socio-economic planners interested in the 
recent development In the field of education. 

4. OBJECTIVES ; 

The chief objectives of this presesslon, offisred for the first time, are these 

(1) to provide a method of assessing the value of Operations 
Analysis to operational and resource allocation problems 
In elementary and secondary education; 

(11) to provide the necessary computer and mathematical techniques 
to appreciate and understand OperatI ons Analysis and Syr.tems 
Approaches to educational planning and administration; and 

(111) to disseminate up-to-date findings In the applications of 
Operations Analysis to education. 
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SCHKtXJLE (ACTUAL) 



DAY 1; SATURDAY. JANUARY 3 0. 1971 



SESSION 


#0: 


8: 


45 - 


10:00 


a .m . 


STAFF: final stages of preparation. 


SESSION 


#1: 


10, 


.30 - 


12;00 


noon 


OPENING OF SESSION. INTRODUCTION OF 
STAFF AND PARTICIPANTS. NATURE AND 
. PHILOSOPHY OF OPERATIONS ANALYSIS. 






12 


:00 - 


1 :30 


p.m. 


LUNCH BREAK 


SESSION 


#2: 


1 


:30 - 


3:00 


p.m. 


REVIEW OF MATHEMATICAL TECHNIQUES. 






3 


:00 - 


3:30 


p '. m . ' 


COFFEE BREAK 


SESSION 


#3: 


3 


:30 - 


5:00 


p.m. 


CASE STUDIES OF OPERATIONS ANALYSIS. 






8 


:30 - 


10:00 


p .m. 


ON-LINE COMPUTER INTERACTION AND 



VOLUNTARY WORKSHOP AND BOOKSHE.'.F 
STUDY. 



DAY 2; SUNDAY. JANUARY 51. 1971 
SESSION #4: 10:30 - 12:00 noon 



SESSION #5: 



SESSION #6: 



12:00 
1 :30 
3:00 
3:30 



1 :30 p.m. 
3:00 p.m. 
3:30 p.m. 
5:00 p.m. 



SPECIAL- ELECTIVE SESSION FOft ADDITIONAL 
REVIEW OF MATHEMATICAL CONCEPTS. 

LUNCH BREAK 

LINEAR PROGRAMMING AND MPS/360, 
COFFEE BREAK 

LINEAR PROGRAMMING IN VOCATIONAL- 
TECHNICAL EDUCATIONAL PLANNING. 



8:30 - 10:00 p.m. 



LINEAR PROGRAMMING WORKSHOP. 
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in • 1 s 


a • III* 


LINEAR PROGRAMMING IN TEACHER SALARY 
SCHEDULES, ' 




10:15 - 


10:30 




COFFEE BREAK 


SESSION #8: 


10:30 - 


12:00 


noon 


SHORT-RANGE PLANNING MODELS. 




12:00 - 


1 :30 


p«m« 


LUNCH BREAK 


SESSION #9: 


1:30 - 


3:00 


p •m« 


MANAGEMENT INFORIMTION SYSTEMS. 




3:00 - 


3:30 


p •m« 


COFFEE BREAK 


SESSION #10: 


3:30 - 


5:00 


p 


SIMULATION TECHNIQUES. 



MID-SESSION EVALUATION (QUESTIONNAIRE) 
8:30 - 10:00 p.m. COMPUTER WORKSI^OP SESSION; BOOKSHELF OPEN. 



DAY 4: TUESDAY. FEBRUARY 2. 1971 
SESSION #11: 9:00 - 10:15 a.m. 

10:15 - 10:30 a.m.- 
SESSION #12: 10:30 - 12:00 noon 

12:00 - 1:30 p.m. 

SESSION #13: 1:30 - 3:00 p.m. 

3:00 - 3:30 p.m. 

SESSION #14: 3:30 - 5:00 p.m. 

5:30 - 7:00 p.m. 
9:00 - 10:00 p.m. 



APPLICATIONS OF OPERATIONS ANALYSIS 
TO PERFORMANCE CONTRACTING. 

COFFEE BREAK 

MATHEMATICAL PROGRAMMING ^^DELS IN 
EDUCATIONAL PLANNING. GROUP DISCUSSION. 

LUNCH BREAK 

APPLICATIONS OF SIMULATION. 
COFFEE BREAK 

APPLICATION OF MONTE CARLO ANALYSIS TO 
USE OF TEACHER SUBSTITUTES. 

STAFF'S COCKTAIL RECEPTION FOR PARTICIPANTS, 

COMPUTER WORKSHOP SESSION. 
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DAY 5: WR DNESOAY. FEB R UARY 3. V97-.1 . l 
^. I 

SESSION #15: 9:00 - 10:15 a.m. MICRO-ECONOMI C THEORY. . | 

10:15 - 10:30 a.m. COFFEE BREAK j 

SESSION #16: 10:30 - 11:30 a.m. COMPREHENSIVE PLANNINr,. ' 

AERA QUESTIONNAIRE EVALUATION ■ ; 
CLOSING REMARKS BY INVITED SPEAKER. 

1 

CLOSING REMARKS BY STAFF. { 

12:30 END OF SESSION., j 



•1 

•1 • 

i- 

\ . 

I 



I? 




6, PARTICIPANTS ; 

Tho following tables summarize Information about the 40 participants, on 
a number of selected variables. This Information was collected prior to 
the pre-session on the standard AERA pre-session Application Form, 

TABLE I 



AGE OF PARTICIPANTS 



AGE FREQUENCY 
20-24 

25-29 2 

30-34 14 

35-39 16 

40-44 9 

45-49 5 

50-54 2 
55+ 



TABLE 2 
SEX OF PARTICIPANTS 

SEX ■ FREQUENCY 
MALE . . 40 
FEMALE 8 



TABLE 3 

GEOGRAPHICAL DISTRIDUTION OF PARTICIPANTS 



STATE FREQUENCY 



Arizona 


2 


Cal i forni a 


4 


Florida 


4 


Georgia 


2 


1 11 inois 


6 


Iowa 


1 


Kentucky 


1 


Maryland 


2 


Michigan 


8 


Minnesota 


3 


Misslssl ppi 


1 


New Mexi co 


1 


New York 


5 


Ohio 


2 


Pennsylvania, 


3 


Tennessee 


1 


Wl scons 1 n 


2 



TABLE 4 



QUESTION; HAVE YOU ATTENDED AN AERA PRESESSION IN THE PAST? 



REhtY 
Yes 
No 



FREQUENCY 
16 
32 



TABLE 5 

EDUCATIONAL BACKGROUN D OF PART ICIPANTS 
YEAR DEGREE RECEIVED: 



MASTER'S 

1937-1950 
1951-1960 
1961-1970 



FREQUENCY 

2. 

15 
24 



DOCTORATE 

1937-19bO 
1951-1960 
1961-1970 

LXfTCTClJd'JVl) 



FREOUEfJCY 



/I 
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TABLE 6 




EMPLOYMENT BACKGROUND OF 


PARTICIPANTS 




FREQUENCY 


University or State College 


23 


City School District 


11 


Regional Educational Labs 


8 


State Department of Education 


6 



TABLE 7 

PROFESSIONAL AND SCHOLARLY INTERESTS OF PARTICIPANTS 



A. Approximately how many research 

articles which you have authored . 
alone or Jointly have been accepted 
In a scholarly (refereed) journal? 



None or No Reply 
1-5 
Over 5 



FREQUENCY 
31 
15 
2 



B. In total, about how many research 
articles, theses, or technical 
reports (both published and unpub- 
lished) have you authored alone or 
. Joi nt ly? 



None or No Reply 
1-5 
Over 5 



FREQUEfjCY 
10 
17 
21 



TABLE 7 (cont'd.) 



How many funded (by USOE, NIMH, 
Ford Foundation, or other granting 
agencies) research projects are in 
progress or completed on which your 
name appears as either the first or 
a joint author? 



TABLE 8 

SOURCE OF PAYMENT OF PARTICIPANTS 

FREQUENCY 



FREQUENCY 



None or No Reply. 



1-5 



Over 5 



Personal 



Institutional 



Research Grant 



Not Given 




7, INSTRUCTIONAL AND EVALUATION MATERIALS ; 



A> ^ MIMEOGRAPHED NOTES 

"Social Goals, Educational Priorities, and Dollars: Planning Education 
In the Seventies/' George S» Tracz. 5 pages • 

Graduate Course 1704X: Course Bibliography. "Mathematical Models in 
Educational Planning." George S. Tracz. 6 pages. 

"Review of Mathematical Concepts in Operations Research, Document One: 
Mathematical Models." James F. McNamara. 6 pagps, 

"Revlewof Mathematical Concepts in Operations Research, Document Two: 
Elernents of Matrix Algebra," James F. McNamara. 27 pages. 

"Review of Mathematical Concepts in Operations Research, Document Three: 
Simultaneous Li near Equati ons," James F. McNamara. 15 pages. 

"Review of Mathematical Concepts i n Operati ons Research , Document Four: 
Differential and Integral Calculus," James F. McNamara. 28 pages. 

"Mathematical Progrgmmlng Models in Educational Planning." James F. 
McNamara. 12 pages.. 

"A Systems Approach to Educational Planning." James F. McNamara. 23 pages. 

"Some Comments on Operations Research and Educatjonal Planning Models." 
James F. McNamara. .1.1 pages,. 

"Mathematical Programming Models in Educational Planning." James F. 
McNamara. 50 pages. 

"A Labor Market Information System for State-Local Program PI anni ng> and 
Evaluation in Vocational Education." James F. McNamara. 14 pages. 

"Linear Programming." James E. Bruno. 50 pages. 

"Notes on the MPS 360 Mathematical Programming System." James E. Bruno. 
18 pages. 

"Operations Research: A Missing Link." G.'. Ernest Anderson, Jr. 
Educatlona I Researcher . Vol. XXI, March*" 1970. 3 pages. 

"Computer Helps Redistrict Schools." James C. Green. Educat i ona I Researche r 
•Vol. XXI, March 1970. 2 pages. 
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7. A. (cont'd.) 

"Simulation Modeling in METEP/^ G. Ernest Anderson, Jr. 7 pages. 
"Queue." G. Ernest Anderson, Jr. 6 pages. 

"Adult Resources Flow (ARF-1).*' G. Ernest Anderson, Jr. 12 pages. 
"EDSIM I." G. Ernest Anderson, Jr. 9 pages. 

"Introduction to UMASS Time-Sharing System."" G. Ernest Anderson, Jr. 

19 pages. 

"Short Range Planning for Educational Management." 1. B. Turksen. 

20 pages . 

Exercise: "Simulated Inventory." S. Padro. 

"Micro-Economic Theory - Theory of the Firm:^ An Outline." S. Temkin. 
7 pages . 

"Comprehensive Planning." S. Temkin. 5 pages. 

"An Assessment of the Contribution of Operations Research to Educational 
Planning." Harold Weitz. 25 pages. 

B. REPORTS 

ERIC Clearinghouse on Educational Administration. Management I nf ormati on 
Systems . Analysis and Blbl iography Series, No. 4. University of 
Oregon. Eugene, Oregon: 1970. 

. . Models for PI anininq . Analysis and Bibliograohy 

Series, No. 5. University of Oregon. Eugene, Oregon: 1970. 

. Models for Rational Decision Making . Analysis and 

Bibliography Series, No. 6. University of Oregon. Eugene, Oregon: 1970. 

Arnold, Walter M. Vocational, Technical and Continuing Education in ^ 
Pennsylvania . A report to the Department of Public Instruction and the 
Pennsylvania State Board of Education. 1969. 

McNamara, James F. A Mathematical Programming Model . Harrisburg, Pa.: 
Pennsylvania Department , of Educati on, 1970. 

McNamara, James F. , and Franchak, Stephen J. Planning Vocational Education 
Programs in Pennsylvania . Harrisburg, Pa.; Pennsylvania Department 
of Education, 1970. 

Temkin, Sanford. A Comprehensive Theory of Cost-Effectiveness . Philadelphia,, 
Pa.: Research for Better Schools, Inc., 1970. 
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/ERA 1971 Research Training Sessions 
Evaluation by Participants 

Mamc AC QcccTAKJ. OPERATIONS ANALYSIS TECHNIQUES IN EDUCATIONAL PLANNING 
IWlb Uh OtbblUIN. ADMINISTRATION 

Director: ^'^^^ '^^^ ^'^^^ ^* ^^^^^ 



NUMBER OF RETURf^lS = 32; SUMMARIES IN % (ROUNDED OFF) . 

I A. To what extent did the re I at i ve avai lab i I i ty or unavai lab i I i ty of books 
and journals interfere with or promote your attempts to master the con- 
tent of this session? 



NONE ^ 



LITTLE 25 



SOME 20 



IB. To what extent did reproduced materials given to you by the staff Improve 
matters? 



I 

■4 

I 
■■■a 

•a 



HELPFUL 



55 



VERY HELPFUL 



45 



2A. Did you feel that you lacked a "place to work", either alone or in smal 
groups? 



YES 



20 



NO 



64 



No COMMENT 



I 
I 

i 



1 



2B. Was your room satisfactory? 

YES ^68 NO 26 



No COMMENT 6 



1 



ERIC 



3A. Which features of the meeting rooms were inadequate or not conducive to 
learning? 



BLACKBOARDS 39 . 
SIZE 19 



LACK OF OVERHEAD 
PROJECTOR 



FURNISHINGS 



13 



LIGHT_ 
AIR 



55 



MISCELLANEOUS. 



26 



48 



3B. Which features were especially facilitative in the same regard? 
LIGHT 6 AIR 3 



SOUND 



42 



FURNISHINGS 13 



CONVENIENCE 



13 



No COMMENT 32 



I 



I 

I 
I 

I 
I 

1: 

i 



mm 



Ms 



/ 
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NUMBER OF RETURNS = 32; SUMMARIES IN % (ROUNDED OFF). 
/".A. Wa<> five days too Icnn a period to leave your work at home for the Durccoc- 
of attending thio session? 



YES 



26_ 



NO 



JA. 



4b. Was five days too short a period in which to loarn much of the content of 
this session? 



YES 



38 



NO 



62 



5A. Were you allowed enough time in which to pursue activities of your own 
choosing? 



YES 



48 



NO 45 



No COMMENT 



5B. Would you haVe preferred not to meet in the evening after dinner? 

YES 48 NO 39 No COMMENT 1 3 



5C. Would you have preferred more or fewer meetings per day than there actually 
were or was the number of meetings per day -agreeable to you? 



FEWER 



ENOUGH 84 



MORE 10 



6A. Were the individual lectures too long to sit and listen or take notes? 

YES 16 NO 84 

68. Were the lectures scheduled in an appropriate sequence? 

YES 42 NO 58 



7. bid you have suf f ici ent opportunities to interact with other pcirtici pants? 



YES 



61 



NO 



39 



—I 



8A. Were the instructors too inaccessible or unapproachable so that you did not 
get the individual attention that you desired? 



YES 



16 NO 74 



No COMMENT 10 



SB. Was it helpful to have graduate student assistants present? . 

YES 42 NO 32 No COMMENT 26 



ERIC 




■77- 



- 14 - 

3 



NUMBER. OF RtTUW^S = 32; SUMMARIES IN % (ROUNDED OFF). 



YESJ 42 NO ^ 



9B. Did the content of the lectures and readings presuppose less previous 
training than you had? 

YES 32 NO 68 



YES 91 NO 9 



12. Were you disappointed .in any way with the group of participants? 
YES 36 NO Si 



YES 68 NO 32 




Ifl 



I 



9A. Did the content of the lectures and readings presuppose r.ore provious (I 
training than you had? " 



10. To what extent was the content of the lectures and readings relevant to 

what you hoped to accomplish during the session? /;c| 

SOME 55 MUCH 42 No COMMENT 3 || 



MA. Were'the lectures stimulating and interesting? 

YES(usually, or somewhat) 94 VERY 6 |^ 

MB. Were the lecturers competent to speak on the subject assigned them? ^ 



1 



lie. Were the lecturers well prepared? 

YES 91 NO 9 i 



■^4 



i 

1 



13. If you had it to do over again would you apply for this session which you 
completed? 

YES 71 . NO 29 



have just completed? - 

I 



m 



14. If a session such as this is held again would you recommend to others like .|| 
you that they attend? 



15. Do you anticipate maintaining some sort of contact with at least one of the 
session staff? 



YES 77 NO "23 M 
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NUMBER OF RETURNS = 32; SUMMARIES IN % (ROUNDED OFF). 

16. Do you feel that AERA is making an important contribution to education 
by sponsoring sessions such as thjs one? 

YES_ SO N O - 3 . 



17, Do you fee) that anything has happened during these f ive-'days to make it 
more likely that you vn I I leave your present position of emp loyment? 

YES 16 NO 



18. Is it likely that you will collaborate in research with someone else, 
attending this session (other than those you already were likely to 
col laborato with.)? 

YES 29 NO 58 



19. Do you think that the staff should feel that It has accomplished Its ob- 
jectives during this five-day session? 

YES^ 62 NO 13 
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MID-SESSION EVALUATION QUESTIONNAIRE 



KEY: SA (Strongly Agrooi^ A (Agree), 
SD (Stronqiy Dir.Dgrci3), NA^ <No 
PLEASE CIRCLE YOUR CHOICES. 

NOTE ; NUMBER OF REPLIES = 38 

!• The object I vos of this program 
were clear to me 



2, The objectives of this program 
were not rsaS Istlc 



3. The pai"ttclpr.nts accepted the 
purposes of this program 



4, The objectives of this program 
were not the same as my 
objectives 



5. I have not loarned much new 



6, The materia 5 presented seemed 
valuable to me 



7^ I could have learned as much 
by reading a book 



( Undecided), 0 ( Disagree), 
wer) , 



IN PERCENTAGES (ROUNDED OFF)., 



SA 


A 


? • 


D 


SD 


NA 


5 


38 


n 

o 






X 


SA 


A 


7 


D 


SD 


NA 


3 


16 


22 


27 


8 


24 


SA 


A 


7 


D 


SD 


NA 


0 


22 


32 


19 


5 


22 


SA 


A 


7 


D 


SO 


NA 


3 


24 


14 


38 


5 


16 


SA 


A 


7 


D 


SD 


NA 


3 


24 


3 


37 


30 


3 


SA 


A 


7 


D 


SD 


NA 


19 


60 


11 


8 




2 


SA 


A 


7 


D 


SD 


KA 


5 


12 


12 


46 


22 


3 



6. Possible solutions to my problems SA A ? D SD NA 

are not being considered 3 27 8 35 11 17 

9. The Information presented was SA A ? D SD NA 
too olomentary 0 13 3 43 38 3 

10. The speakers reolly know thoir SA A ? U ZU KA 
subjects 41 46 !; 0 0 8 

11. I was r>tirauIotc>d to think about SA .A Y 0 SO NA 
tf^o ioptcs presontod 24 57 .3 ,13 0 



MK)-SESSION EVAlilATICM QUHSTiOMNAf Rg (continued) 



NOTE: NUMBER OF REPLIES = 38 

12. V/e workod toq«thor wo if as a 
group 

13. Tho group discussions v/ore 
oxcel lent 

14. Thore was HttSe tln^ ror 
Informaf conversation 



15* I had no opportunity to express 
rrty ideas 

16. I realJy felt a part of this 
group 



17. My time was woM spent 



t8» The program mot my expectations 

19. Too much tlnve was devoted to 
trivial matters 



20. The Information presented was 
too advanced 



21. The contont was not readily 
appllcabSe to much research 
In education 



22. The Assistant was very helpfu! 



23. Theory was not related to pracM 

24. The schedule should have been 
more flexible 



IN PERCENTAGES (ROUNDED OFF). 



SA 


A 


? 


D 


SD 


NA 


8 


19 


27 


27 


8 


n 


SA 


A 


? 


D 


SD 


NA 


9 


16 


24 


35 


3 


13 


SA 


A 


? 


D 


SD 


NA 




24 




51 


8 




SA 


A 


? 


D 


SD 


NA 


2 


5 


8 


57 


16 


12 


SA 


A 


? 


D 


SD 


NA 


5 . 


. 43 


16 


27 


5 


4 


SA 


A 


? 


D 


SD 


NA 


2 


54 


15 


24 


0 


5 


SA 


A 


? 


D 


SD 


NA 


5 


32 


30 


19 


3 


n 


SA 


A 


? 


D 


SO 


NA. 


0 


15 


13 


54 


8 


10 


SA 


A 


? 


D 


SD 


' NA 




24 


1 1 


50 


8 


5 




f\ 


<> 


U 


en 




3 


19 


10 


44 


19 


5 


SA 


A 


? 


D 


SD 


NA 


10 


35 


36 


0 


5 


14 


SA 


A 


? 


D 


SD 


NA 


5 


16 


5 


60 


8 


6 


SA 


A 


? 


0 


SD 


NA 


3 


27 


8 


54 


5 


3 
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9. DIRECTOR'S EVALUATION ; % 



(A) Some open-ended replies by the participants to the AERA Evaluation 
Questi onnai re ; 

Q. IB. To what extent did reproduced materi a Is given to you by the staff 

Improve matters? 
A. McNamara supplied us with excellent materials. 



■J 



Q. 3A. V/hich features of the meeting rooms were inadequate or not conducivie 

J 

to learning? ;?< 

I 

A. Cold temperatures at one session. (Reference to Consolidated i| 

Edison Utl llties.) | 

Q. 6B. Were the lectures scheduled in an appropriate sequence? • ::| 

. - ■ I 

A. Originally, .they were not, but directors did make adjustments in :<| 

schedule. . I 
Q. 8A. IVere th$ Instructors too Inaccessible or unapproachable* 50 that you 

did not get the Individual attention that you desired? I 

■ * } 
A. All Instructors were Interested and they acted just like us common | 

folk. 1 

1 

Leaders were interested in "humans" often a quality missing In | 

I 

research types. : | 

k 

Q. 10. To what extent was the content of the lectures and readi nps .re levant | 
to what you hoped to accomp I Ish. durl ng the session? | 
A. The review was Invaluable and several new ideas were presented. | 

Q. 16. Do you feel that AERA is making an important contribution to education | 

by sponsoring session such as this one? | 
A. Semiannually would be nice. | 

Q.* 19. Do you think that the staff should feel that it has accomplished ^ | 

Its objectives during this five-day session? I 
A. Yes, as a first effort. I 

i 
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GENERAL COMMENTS ; 

- the ''Happy Hour" should be planned for the beginning. 

- gave us an opportunity to not only develop our own crude model, but 
to try it out, with staff assistance, on the teletype. 

- the descriptions of the sessions as presented in the AERA Newsletter 
should be much more detai led in order to assist people in selecting 
an appropriate session. 

- this Is a very poor eva I uati on instrument (AERA's) with many inconsistencies 
• and positively weighted questions, (e.g., #11A.) 

DIRECTOR'S COMMENTS : 

Two Important considerations must be recalled. This session was the first 
of. Its kind held anywhere, and the unsettled funding situation-persisted 



for a long time. 

The session was particularly satisfying to the directors and the staff because 

1. It brought together a large number of researchers (who previously 
had never attended an AERA pre-session) from many distant places 

and diverse employment backgrounds. 

2. It allowed the participants to establish a dialogue and over- 
come their pre-defined positions in the locus of decision-making. 

3. It assisted the participants in developing and refining their 
quantitative skills of systems analysis. 

4. I t poI nted. out to the staff possible Improvements in setting 
the format for a future undertaking of this type. 




• DIRECTOR'S COmENTS ; (cont'd.) 
Taking this report Into an overall perspective, the directors are strongly 
Justified to feel, as did the participants, that this first session on 
"Operations Analysis Techniques in Educational Planninq and Administration" 
went a long way to meet its objectives successfully. 
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1, Title: Individual Differences, Learning and Instruction 

2, Staff: Frank H, Farley (Director), University of Wisconsin 

Thomas J, Shuell, State University of New York at Buffalo 
Richard E, Snow, Stanford University 

Arthur R, Jensen (two days). University of California, Berkeley 
Lauren B, Resnick (one day). University of Pittsburgh 
Robert C. Calfee (one session), Stanford University 
Wallace L. Mealiea (one session). University of Wisconsin 

3, General Description: 

The content of this presession was heavily in the direction of what 
is presently known about individual differences in learning and instruc- 
tion, what are the best methodologies for the solution of the problems 
in the area, what are the implications of the knowledge and methodology 
for educational research, and what are their use in educational practice. 
• The scientific winnowing and sifting of the research evidence was emphasized 
and the most recent best research was included. 

The individualization of learning and instruction is a major goal 
of American education. In addition, the role of individual differences 
in learning is a basic concern in the scientific study of learning and 
instruction. Where the former is concerned, a number of individualized 
learning and instruction programs have been developed in recent years and 
disseminated in schools. The psychology underlying some of these is 
often unclear, and their efficacy relative to other approaches is often 
unknown. Wl}ere the latter is concerned, the inclusion of individual 
difference terms in the development of learning and instructional theories 
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remains one of the most difficult and least satis factorially handled 
problems in theory construction. 

This presession considered the basic modes of Individualizing instruc- 
tion, the merits of their psychological bases, and the direction research 
into individualization should most fruitfully take. In addition, opti- 
mal ways of including individual difference considerations in the develop- 
ment of learning and instructional theories were dealt with throughout 
the presession. 

4. Objectives: 

The general objective was to attain theoretical and research sophis- 
tication in the study of individual differences in learning and ins true- 
tion, and in the individualization of school learning. The specific 
objective was to conceptualize and design a research study or program of 
studies directed at the solution of a significant problem(s) in this area. 

5, Schedule: 

First Day: Morning: 



Introduction of staff. Introduction to the session. 



Discussion of session objectives. Entering level test. 



Historical overview of the field. Theoretical models 



and issues in the study of individual differences and 



learning. 



Afternoon: 



Individual differences in retention and forgetting. 



Ability, fast versus slow learners, and forgetting. 



Interactions of presentation mode and ability. 



. Second Day: Morningt 



Discussion of previous day's sessions. 




Biological and psychophysiological approaches. 
Teplovian analyses. Intrinsic individual differences. 
Motivational-attentional approaches to individual dif- 
ferences in learning and memory. 
Afternoon: 

Continuation of above. Outline of a heuristic for 
studying motivational-attentional-personality individual 
difference variables in learning and memory. 
Introduction to aptitude x treatment interaction re- 
search. 'Measurement considerations. 
Third Day: Morning: 

Discussion of previous day's sessions. 

Aptitude X treatment interactions continued. Methodologi 

cal issues. 

Afternoon: 

Continuation of above. 

/ 

Representations of individual differences in learning. 
Woodrow tradition. Methodological issues. The measure- 
ment of change. 

Gain scores; residual gain. Learning and ability. 
Special interest groups meet with individual staff 
members for discussion session. 
Fourth Day: Morning: 

Discussion of previous day's sessions 

Consideration of heredity and environment in determina- 
tion of learning performance and intelligence. 
Methodological issues. Sex differences. Socio-economic 
status factors . 
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Afternoon: 

Hierarchies of ability and school instruction. 
Intelligence and thinking versus learning. 
Evening: Special Session 

Some methodological approaches to subject x treatment 
interactions, with an emphasis on applications to pre- 
reading skills and reading acquisition. 
Fifth Day: Morning: 

Discussion of previous day's sessions. 

The individualization of learning and instruction. 

Afternoon: 

Continuation of above. 

Division into two groups (1) continuation of above 
and (2) behavior modification and individual differences. 
Critical individual discussions of research questions and/ 
or designs concerning individual differences, learning 
and instruction which have been generated by participants. 

6. Participants: 

Sixty-two individuals applied to the AERA Presession Individual Dif- 
ferences, Learning and Instruction. The staff had set a limit of 50 
participants in the published announcement of the presession. However, 
this limit was not rigidly adhered to. A total of 54 persons were ac- 
cepted for the presession. It was felt at that time that many more would 
have made the total group unmanageable. Those persons rejected were 
mainly very late in applying, after the staff had agreed to cease any 
further admitting. Of the 54 accepted applicants, 44 actually appeared 
at. the presession. At the conclusion of the presession, the staff felt 
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that a group considerably larger than the 44 could be handled without 

difficulty with the format employed. 

7. Instructional and Evaluation Materials: 

Participants were assigned three seminal readings to be completed 
prior to attendance at the presession. These were selected exclusively 
from R, M. Gagne^ (ed.) Learning and individual differences . Columbus, 
Ohio: C. E. Merrill. This book was chosen because of its central 
character in the field, and because it was expected that if all readings 
came from one source there would be more chance of the participants 
completing each of the readings prior to the presession. The chapters 
assigned were: 

Glaser, R. Some implications of previous work on learning and 

individual differences. 
Cronbach, L. J. How can instruction be adapted to indiviSual 

differences? 

Jensen, A. R. Varieties of individual differences in learning. 
A brief 12 item achievement test based on these three chapters was given 
all participants in the first hour of the presession. 

An approximately 200 item bibliography prepared by the Director on 
the topic of Individual Differences, Learning and Instruction was given 
to all participants the first morning of the presession. A library of 
reprints was maintained, with every item in the bibliography being 
represented in the library by from two to ten copies. Participants 
could borrow these materials for daily or overnight use. In addition, 
a number of books and individualized instruction program manuals were 
available in this library. A typewriter was made available for partici- 
pant use. A list of participants and their research interests was provided 
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each participant, as well as a program outline for presession. 

A number of handouts were provided during the presession. These 

were: 

- General 

1. A complete bibliography of Project PLAN, plus assorted brochures. 

2. A complete bibliography of Individually Guided Educatign in the 
Multiunit School. Also a complete bibliography of "Individually 
Guided Motivation." 

Specific 

1. Farley presentations: Farley, F. H. , & Severson, H. H. 

"The stability of individual differences in the strength 
and sensitivity of the nervous system." 

2. Snow presentations: "Aptitude and instructional methods: 

the search for interactions." 

3. Jensen presentations: "Do schools cheat minority children?" 

In addition, a 200 page review by Jensen will be distri- 
buted to all. participants when it is completed in May or 
June of 1971. 

4. Resnick presentation: "Implications of individual differences 

for the design of instructional environments. Topics for 
discussion." 

All formal sessions made extensive use both of slides and overhead pro- 
jection. 

Besides the printed material noted above, seven films and/or film 
cartridges on individualized learning and instruction programs were 
available for screening by participants. One special session for partici 
pants interested in these programs was provided, with the films being 
shown for group discussion and analysis. 
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8« Evaluation and Test Results: 

The brief (12 item) "entering level test" based on the three read- 
ings to be completed prior to the presession yielded the following re- 
sults: of the 37 perso:as who returned the questionnaire, the mean score 
was 6,4, suggesting a fairly low- level of achievement for the three, (rela- 
tively straightforward) readings , Participants were asked to indicate 
whether they had completed the assigned readings prior to arrival at the 
presession. Of the 37 respondents,. 26 or 70,3% had read all or portions 
of the readings, while 11 or 29.7% had read none of the assigned material , 
The mean test performance of the former was 7*1, while that of the latter 
was 4,8 (t = 3.07, p < .01). 

The objective of designing a study was optional for all partici- 
pants, as a large number indicated they were less interested in develop- 
ing a study or research program at that time than in being exposed to 
the most recent developments in the area, with an objective of consolidat- 
ing this knowledge after the presession with perhaps the development of 
research or enrichment of their current research efforts, coming then. 
Accordingly, only a few participants discussed with the staff specific 
research designs developed during the session. These were judged satis- 
factory by the staff, and in some cases, continued interaction with staff 
has taken place following the presession. 

On the last afternoon of the presession the anonymous session evalua- 
tion questionnaire developed by AERA Central Office in cooperation with 
the directors of the individual sessions was distributed. The responses 
of the participants were generally quite favorable. A slight sequencing 
problem where lectures were^concerned was apparent from the responses to 
one of the 30 questions asked. One or two lectures had to be presented 




somewhat out of a logical sequence because the speakers vere not avail- 
able at any other time. 
.9. Director's Evaluation: 

The presession seemed to be veil received by participants and seemed 
to represent a successful attempt to pull together a dynamic and currently 
very active but diverse area of. educational research. On the basis of 
viewpoints expressed by participants, a slightly increased emphasis on 
the area of aptitude x treatment interactions might be desirable in 
future comparable sessions. 
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/ERA 1971 Research Training Sessions 
Evaluation by Participants 



M o INDIVIDUAL DIFFERENCES IN LEARNING AND INSTRUCTION 
jWIE of SFSSION: . 

Dl RECTO 



IA. To what extent did the relative availability or unavailability of books 
and journals interfere with or promote your attempts to master the con- 
tent of this session? 

NONE 55 % L I TTL E 15% ' SOME 30% 

IB. To what extent did reproduced materials given to you by the staff improve 
matters? 

• HELPFU L 90% VERY' HELPFUL 10% 

*2A. Did you feel* that you lacked a "place to work", either alone or in small 
groups? 

/ 

YES 20% N O >0% . No COMMEN T 10% 

2B. Was your room satisfactory? 

YES 65% N O 25% No COMMENT 10% . 

3A. Which features of the meeting rooms were Inadequate or not'conduci ve to 
learning? 

BLACKBOARDS 95% LACK OF OVERHEAD 

■ . PROJECTOR 0% 



S i ZE 10% . 

" FURNISHINGS 0% 



LIGHT 5% . 

M i SCE LLANEOUS_;^ 20%_ 

AIR 65% . 



3B. Which features were especially faci I i tati ve in the same regard? 

--L-I-6HT ~ 0% AIR_ 0% 

SOUND 10% FURNISHINGS . 10% 



CONVENIENCE 25% ... No COMMENT 60% 
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<^fA. Wds five days too long a period to leave your wori< at home for the purpose 
of attending this session? 

YES < 65% NO 35% 



YES 10% NO ,90% 



5A. Were you allowed enough time in which to pursue activities of your own 
choosing? 

r YES 65% NO 30% No CaMMENT 5% 



5B. Would you have preferred not to meet in the evening after dinner? 

YES ' .30% N O 45% No COMMENT 25% 

5C. . Woufd you have preferred more or fewer meetings per day than there actual ly 
were or was the number of meetings per day agreeable to you? 

FEWER 5% ENOUGH 90% MORE 5% 



m 



4B, Was five days too short a period in which to learn, nuch of the content of . .vj^ 

this session? M 



6A* Were the individual lectures too long to sit and listen or take notes? | 

YES 15% N O 85% ' | 

6B. Were the lectures scheduled in* an appropriate sequence? f| 

YES NO 60% I 

3 

s, 

1, Did you have sufficient opportunities to interact wi.th other participants? | 
YES 85% ■ NO 15% | 



■6A. Were the instructors too inaccessible or unapproachable so that you did not 4 

get the individual attention that you ' desi red? . ' | 

YES ^ ■ N O 85% . No COMMENT | 

8B. Was it helpful to have graduate student assistants present? |' 

YES NO No COMMENT ; | 
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9A.' Di.d the content of the lectures and readings presuppose more previous 
training than you had? 



YES 



20% 



NO 



80% 



9B. Did the content of the lectures and readings presuppose less previous 
training than you had? 



YES 



20% 



NO 



80% 



■m 

■■ix 

1 

1 
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10. To what e^Ktent was the content of the lectures and readings relevant to 
what you hoped to accomplish during the session? 

MUCH 



SOME 



No COMMENT 



I- 



I 
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lA. Were the lectures stimulating and Interesting? 

YES(usually. or somewhat) 5^^:. . VERY_ 



5% 



IB. Were the lecturers competent to speak on the subject assigned them? 

' 95% 5% 



YES 



NO 



lie. Were the lecturers well prepared? 



YES 



70% 



NO 



.30% 



\2, Were you disappointed in any way with the group of, participants? 

YES 50% NO . .50% . 

13. If you had it to do over again would you apply for this session which you 
have just completed? 



YES 



85% 



NO 



15% 



14. If a session such as this is held again would you recommend to others like 
you that they attend? 



YES 



80% 



NO 



20% 



15. Do you anticipate maintaining some sort of contact with at least one of the 
session staff? 



YES 



80% 



NO 



20% . 



ERIC 
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Do -you feel thar AERA is making an important contribution to education 
by sponsoring sessions such as this one? 

YES 85% NO 15% 



Do you feel that anything has happened during these five days to make it 
more iikqly that you wi I I leave your present position of employment? 

YES 15% . NO 85% 



Is it likely that you wl I I col laborate in research with someone else 
attending this session (other than those you already were likely to 
col laborate with)? 



YES 



20% NO 80% 



Do you think that the staff should feel that it has accomplished its ob- 
jectives during this five-day session? 

YES ;50% N O 50% ' 
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PARTICIPANT ROSTER 

Individual Differences, Learning and Instruction - AERA Research Training Session 



AUSTIN,. Gilbert R. 
Secretary's Office 
Health, Education and Welfare 
Washington, D.C. 20201 

Research on Compensatory education. 

.BARTON, Grant E. 

Instructional Research and Development Dept. 

Brigham Young University 

Provo, Utah 

Development of systematically designed and 
validated instructional packages. 

BERNSTEIN, Norma T. 

California State College 

Long Beach, California 

Curriculum development and individualiza- 
tion of instruction in area of language 
and thinking 

BILSKY, Maxwell G. 
Madison College 
Harrisonburg, Virginia 22801 

Motivation and learning from students 

and instructors viewpoint. 

BORICH, Gary D. 
Institute for Child Study 
University Schools 
Indiana University 
Bloomington, Indiana 

Learning and individual differences 

BORTNICK, Robert M. ' 

University of Chicago and 

Glen Ellyn Elementary School District 

Glen Ellyn, Illinois 

Criterion references testing, cognition. 

BRODY, Helen, M. — ^ 
Dept. of. Student Personnel 
Queensborough Community College 
City University of New York 
Bayside, New York 11364 

Educational ~ change, personal development 

change. 

BRYANT, Norman D. 
Teachers College 

Columbia University A ^ 

New York, New York 10027 

Learning disabilities and educationally 
relevant neuropsychological process. 



BUCK, Elizabeth B. 

School of Education 

University of California 

Los Angeles, California 

Motivation, affective measures, opera- 
tions research approach in education. 

CLARK, Richard M. 
Dept. of Educational Psychology 
State University of New York 
Albany,. New York 12203 . 

Child development, school learning. 

COFFING, David. G. 

Educational Media, Technology and Systems 
School of Education 
University of Massachusetts 
Amherst, Massachusetts 

Paired-associate learning, pictorial 

versus symbolic representation, character-. 

istic based learning i .e. , aptitude - 

treatment interaction. 

CONNOLLY, John A. 

American Institutes for Research 

.Washington Office 

The measurement and instructional treat- 
ment of individual differences in learning 

EVANS, Ross A. 

Research and Demonstration Center for the 
Education of Handicapped Children 
Teachers College 
Columbia University 
New York, New York 10027 

Basic and applied learning research ! 

with the mentally retarded. 

FULD, Paula A. 

Dept. of Foundations, Psychology and Services 

School of Education 

Fordham University 

New York, New York 10023 

Verbal learning, individual differences, 

learning disabilities. 

GILL, Noel C. ' 

Psychology Department 

Illinois State University 

Normal , Illinois 

Individualized instruction, attitude 
formation, group work and communication 
skills. 
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GRIMALDI, Joseph 

Psychological and Educational Testing and 
Research 

Marymount College 

Tarrytown, New York 

Neuropsychology, personality measurement, 
psychophysiological aspects of emotion. 

6USTAFS0N, Kent L. 
Instructional Media Center 
Michigan State University 
East Lansing, Michigan 48823 

Design and development of validated 

instructional systems 

HARRIS, Elizabeth 
Computer Services 
Illinois State University 
Normal, Illinois 

Individual differences related to learning 

experimental design. 

HAWKES, Ellen G. 

Public School System 

Newtown, Connecticut. 

Curriculum development in social studies 
curriculum coordination K-12*, individua- 
lized instruction, programmed instruction 
in cognition, reflective teaching, dif- 
ferential learning style diagnosis. 

HONEYCUTT, Joan K. 

Dept of Early and Middle Childhood 

College of Education 

The Ohio State University 

Columbus, Ohio 

Learning theories and their application. 

JACOBS, Paul I. 
Ferkauf Graduate School 
Yeshiva University 
New York, New York 10003 

, Experimental manipulations that affect 

, cognitive development. 

JENNINGS, Bettye Lea 
Elementary and Special Education 
College of Education 
Michigan State University 
East Lansing, Michigan 48823 
curriculum development. 

KINGSLEY, David C. 

Learning Institute of North Carolina 
Durham, North Carolina 27701 

Measurement of individual differences' 



LEE, Key T. 

Dept. of Educational Psychology and 

Foundations 
University of Northern Iowa 
Cedar Falls, Iowa 50613 

Teacher education. 

LEE, Moon Yong 
College of Education 
Michigan State University 
East Lansing, Michigan 48823 
Motivation 

LEONARD, W. Patrick 
Instructional Materials Center 
Temple University 
Philadelphia, Pennsylvania 
'Curriculum development, teacher education 

■ ■ y 

LOVELESS, Eugene J. 

Dept. of Student Personnel 

Queensborough Community College 

City University of New York 

Bayside, New York 11364 

Individual differences, attitude scale 
construction, measurement of personality 
cognitive styles and orienting behavior. 

LOWE, William T. 

Dept. of Curriculum and Teaching 
College of Education 
University of Rochester 
Rochester, New York 14627 

McLOUGHLIN, William P. 

Dept. of Curriculum and Instruction 

St. John's University 

New York, New York 

Citrriculum development, nongraded school 
individualization of instruction. 

MILLER, George 

Learning Research and Development Center 
University of Pittsburg 
Pittsburg, Pennsylvania 15213 

Curriculum development, instructional 
■ materials design, instructional systems. 

NATKIN, Gerald L. 

Dept. of Education 

Bucknell University ■ 

Lewisburg, Pennsylvania 17837 

Learning and memory, individual differ- 
ences, path analysis. 
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KOSMAC, Leon a 

Eastern Reginal Institute for Education 
Syracuse, New York 

Creativity, individual differences, 

learning. 

KROLL, Herman 

Dept. of Instructional Technology 

School of Education 

University of Southern California 

Los Angeles, California 

Learning theory, mediated instruction, 
experimental design, individual 
differences. 

LANDRUM, Bill 

Charleston County School District 
Charleston, S. C. 

Statistical theory, measurement 

NUMMEDAL, SuSan 6. 

Early Childhood Research Center 

University of California 

Los Angeles, California 90024 

Motivation and cognitive development . 
of young children of various ethnic 
and socioeconomic backgrounds. 



ROELKE, Patricia L. 

Psychology Department 

Emory University 

Atlanta, Georgia 

Language development, particularly 
semantics, relation of cognitive develop- 
ment-concept formation to language develop 
ment application to reading ability-teach- 
ing. 
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in Education 



RUBIN, Rosalyn A. 
Research and Development Center 

of Handicapped Children 
University of Minnesota 
Minneapolis, Minnesota 

Biosocial characteristics, individual 

differences, gifted. 

SCHALOCK, H. Del 
Teaching Research 

Oregon State System of Higher Education 

Monmouth, Oregon 97361 

Human development within the context of 
education, educational models, principles 
of instruction. 
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OHNMACHT, Fred W. 
Dept. of Educational Psychology 
State University of New York 
Albany, New York 12203 

Transfer of abilities at different 



of acquisition, associational structures 
of school related concepts. 



SMITH, Earl P. 
School of Educatio 
University of Virginia 
Charlottesville, Virginia 22901 

Selection and utilization of visual mater 
lals, formulation of objectives,. teacher 
stages characteristics. 



PIPER, Terrence J. 

Dept. of Exceptional Education 

School of Education 

University of Wisconsin 

Milwaukee, Wisconsin 

Motivation, individual learning 
differences, individualized in- 
struction, remediation of learning 
deficits. 

REID, J. Christopher 
Educational Research 
University of Missouri 
Columbia, Missouri 65201 

Human Learning, measurement, 

statistics. 



SPANGENBER6, Ronald W. 
Human Resources Research Organization 
Division No. 2 
Fort Knox, Kentucky 40121 

Message design and learning. 
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SPORB0R6, Anthony 

Curriculum Studies and Research 

Instructional Services 

Wilton Public Schools 
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Individualism in students, divergent i| 
versus convergent productability and its If 
implications for instruction and learninci| 

STARKMAN, Stanley S. * 

Educational Laboratory 

Chicago State College - West Center 

Chicago, Illinois 60624 « 
Instruction, remedial reading, experimenif 
design, communication and classroom inter| 

action. 
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RESEARCH TRAINING SESSION 2 

-1. Title: Multivariate Statistical Analysis in Educational Research 

2. Staff: M. I* Chas. E. Woodson University of California, Berkeley 

R. Darrell Bock University of Chicago 

Neil H. Timm University of Pitts't)ui'g;h 

Jereroy D. Finn , State Univercity of Nev/ York at Buffalo 

Joel R. Levin Uni^versity of Wisconsin 

Robert M. Pruzek State University of New York at Albany 

R. Gnanadesikan ' Bell Laboratories 

3, General Description: 

This session eonsisted of an introduction to the concepts and 
techniques of multivariate analysis, including computer prc^graiiis for 
making the appropriate calculations. Topics included: Univariate — 
Analysis of Variance in Matrix Notation, Hotellings' ir, multivariate 
regression and canonical correlation, multivariate analysis of 
variance, multivariate data reduction techniques, post -hoc tests, 
and other multivariate techniques, with emphasis upon multivariate 
analysis of veirieinee. 

Lectures began with a review of univariate staticticc and tho . , 
expression of univariate statistics in the matrix, formulation 
necessary for multivariate designs. Much of the time there were 
alternative lectures with one designed for relatively advanced 
participants and another designed for those encountering multivariate 
techniques for the first time. 

Although many of the recent advances in statistical inference 
for multivariate problems have not yet become widely used by behavi- 
oral scientists, there is an increasing awareness of the iiaportance 
of multivariate statistieal techniques for behavioral science research. 
Most behavioral science studies involve multiple dependent variables 
and it is desirable to simultaneously use the information from these 
multiple variables in making statistical inferences, llultivariate 
methods bring to such research problems the techniques of ey.porimen- 
tal design, linear estimation of points and intervals, tochniquor; of 
experimental design, linear estimation of point;; cavi ini,f;rv.*il:i, find 
tests of hypotheses which have been so useful in univarJalc niluuLJojKj. 
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Objoctivec: 



The primaiy objectives of this presession were: 

1. To present and interrelate the "basic techniques and concept 
of miativariate statistical analysis and to provide a conceptual 
foitndation to research problems. In many cases this arnoimted to 
introducing the participants to nevr vays of thinking ahout their 



2. To assist the participants in gaining some practical loiow- 
ledge of the use of computers for doing multivariate analysis 
calculations. 



data. 




i 



i c. cptnrraiLF OF T}IE 1971 AERA RESEARCH TRAINING SESSION ON MULTIVARIATE 
I ^' SS?SgS IllJiisS {lEETING AT EDUCATIONAL TESTD^G SERVICE 

General Cedents: Session, will begin at 9:00 ^O^^^^^t'"" 
at ETS;- includo a morning coffee break (10:30 a.m.) , a ^'OOt^-OO brcaK 
?;r iLi? ar.d an afternoon eoffee break (3:00 P---) , v^^^^^^f 
approximately p.m. Evening sessions will be held at the Nassau 

Saturday, 30 January. 1971 (Introduction and Univariate Analysis) 

1. Overview of the Session and Procedures (Woodson). 

2. Welcome from ETS; research at ETS (Messick of ETS). 

3. The Role of Multivariate Methods in Educational Research (Bock). 

U. Elementary Matrix Operations, Matri^ Algebra, Data Matrices, 
and Matrix fonaulation of Designs (Woodson, Pruzek, Levxn) . 

$. Introduction to Coii?)uters, Univariate Exaniples (Finn). 

6. General Multivariate Linear Model: One-sainple, Two-sample (Timm) . 

<;„r,H»,r ^1 Tanuarv 1971 (Two-sample Multivariate Cases, Introduction to 
.Sunday, 31 January, 19a UW^ ^amp^^^ ^^^^^ ^^^^^^ ^^^^^^ development of 

Univariate Cases) 

7. Announcements (Woodson). 

8a. Intuitive Introduction to Multivariate Analysis of Variance (Pruzek) . 
.8b. Exanples: Univariate (Levin). 

9a. MANOVA; One -Way, & Criteria (Timm). ' n 

9b. Two-sample & k-sample, univariate problems (Woodson;. 

10a. Discriminant Analysis, two sample case (Pruzek). 
10b. Post-hoc Tests for Univariate Problems (Woodson). 

11a. Examples (Levin) . - . 

lib. Office Hours (VJoodson) . 

11c. Office Hours (Bock) . . ^ • . ^^^^v,r,^ 

lid. Computer Session: control cards and exercises ^Finn; . 

Monday. 1 February, 1971 (Multivariate Analysis of Variance) 

12. Announcements (Woodson) 

13. Repeated Measures, I (Bock). 
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Iha. MANOVA! Factorial Designs (Timm). 
Ihb. Examples (Finn). 



l^a. Factorial vs Nested Designs, and Post-hoc Tests (Levin), 
l^b. Tour of ETS. 

16a. Regression, Cannonical Correlation, Multivariate Analysis of Covariance (Ttom). 
^ l6b. Examples (Woodson) . 
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17a, Example: Multiple Multivariate Regression (Finn). 

17b, Factor Analysis S& Covariance Structure Analysis, I (Pruzek) • 

l8a. Examples (Levin) • 
l8b. Office Hours (Timm).^ 
I8c, Office Hours (Pruzek). 

I8d, Computer Session: Results f rom MANOVA exercise (Finn). 
Tuesday^ 2 February , 1971 (Special Topics of Multivariate Analysis) 

19. Announcements (Woodson). 

20. Repeated Measures, II (Bock). 
21a. MANOVA: left over topics (Tajnm). 

21b. ExaiT5)le: 2x3 multivariate Design with Post-hoc Tests (Woodson). 

22a. MANOVA: Nested Designs and Block Designs (Timm). 

22b. Factor Analysis & Covariance Structure Analysis, II (Prui^.ek) . 

23a. Some relationships between factor analysis and MANOVA (Pruzek). 
23b. Examples (Levin) . . 

2Ua. 2-JH:-k MANOVA Designs and Planned Contrasts (Pruzek) . 
2Ub. Example: unequal n published study (Finn) . 
2Uc. Tour of ETS. 

25a. Missing data in data reduction problems (Timm). 
2^. Computer Session: Review Regression exercise (Finn). 

25c. Introduction to the Cramer Program for Multivariate Analysis (Chas. Hall of 
25d. Office Hours (Levin). 

Wednesday, 3 February, 1971 (Overview of Multivariate Analysis) 

26. Announcements (Woodson).. 

27. Informal Techniques of Inference, I (Gnanadesikan) . 

28. Methods and Issues in the Analysis of Repeated Measures and the 
Analysis of Covariance (Bock). 

29. Informal Techniques of Inference, II (Gnanadesikan). 

30. Discussion and evaluation of session (Woodson) • 

31. Panol dlncuorjlon (quootilona and comments from the participants wolcomo) : 

. Multlvarlato Ktatiatical Analyoiu in Fxiucatlonal Research (all Instructors). 



Participants: 



Applications were screened and evaluated by the director in 
consultation with the staff. Approximately l68 formal and 25 infoii:;al 
applications were received^ 99 were adjnitted (approximately 20 as 
replacements for cancelation), and' 77 actually attended. 

A priority system was established on admissions. Persons holding 
a doctorate and teaching graduate courses in applied statistics at the 
level of Hays and Kirk were admitted at once. Following that, ixjrsons 
with a doctorate and reporting considerable formal training in applied 
statistics were admitted. Following that, persons without the doctorate 
but- with special interest in these techniques were admitted. Persons 
indicating very little baclcground in statistics were not adioittcd. Tiio 
standards for admission became stricter as the available space filled. 

The participants; on the average , indicated they had an average of 
if. 6 (s=2.8) graduate courses in statistics, 1.8 (s=l.if) graduate courses 
in measurement; taught 2.7 (s=3-8) courses in elementary stastics, 
taught 1.1 (s^l.9) courses on research design, and had published an 
average of 3.7 (s- 5»2) research articles. 

The average level of statistical competency was higher than last 
year. 'Ihis is probably a function of persons with a particular intei^st 
in the techniques becoming aware of the program as well as the selec- 
tivity of admissions. Participants included researchez's in biology, 
medicine, public health, computer science, as well as in psychology 
and education. 

Most participants were from the east, some from the south and 
midwest, and very few from the west. Many of the cancelations were 
from the west and referred to travel funds as a major problem. 

Materials 

A large number of handouts were prepared and used during this ses- 
sion. These handouts, as last year, received a large number of favorable 
remarks. The letter of acceptance to '^participants suggested texts, 
readings, and. concepts the participant should explore as preparation 
for the session. 

Copies of the chapter by Bock and Haggard, some duplicated materials 
by Woodson > and infoimation about ETS and Princeton were mailed to 
participants ^ 

At the session, the following materials were dir.tributed: 

1. Bock, R. D. & Haggard, E. A, T^ie use of multivariate analysis 
in behavioral research, Whitla, D. K. (Ed.), Handboo?. of 
Meas urement and Assessment in Behavioral Sciences . Kcading, 
Mas. . Addison, Wesley, 196d. 

'2. Finn, J. D., Multivarlance; -Fortran Profcrnm for Uni vcfrjalo ' 
and Multivariate Analysis of Variance and Covarlance , {jLute 
University of New York at BulTalo, 1S;67. 
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Timm, N. 11. Prepublication copies of three clzapters on raulti- 
variate statistics. 



k. Pruzck, B. Two in-presc articles and other class notes. 

^. Levin, J. Several duplicated exemiples and other class notes. 

6. Bock, R. D. Chapters k and 5 forthcoming book on raulti- 
variate statistics. 

7. Finn, J. D» Several duplicated examples. 

, 8# Gnanadesikan, R. Bibliography on informal methods. 

9« Woodson, Mf !• C* E. Duplicated lecture: notes, exaraples, and 
bibliography. 

Evaluation and Test Results: 

Short q\iizzes on matrix algebra, linear models and ajialysis of experi- 
mental data were given early in the session. Correlations of rscores on 
these tests and a number of descriptive variables vrere examined. The 
highest correlations (about •5) were with number of statistics courses 
taught and number of research design courses taught. 

. ■' * • 

A number of short quizzes were given during the lectures. These 
were used primarily by the instructor involved and provided a useful 
feedback device, particularly for dealing with the less voc£tl i)articipaiits 
and early in the program. 

Pai'ticipants were encouraged to give their comments and problems to 
the director. Many suggestions were made and many adaptations to the 
program were made. 

An evaluation questionnaire was distributed at the end of the cession. 
Most participants felt satisfaction with the staff and lectures. 

Director's Evaluation: 

The major factor in the success of the research training session on 
•"Multivariate Statistical Analysis" was the excellent staff. Their work 
and high degree of competence combined favorably with their variety of 
interests and approaches. 

The cooperation and assistance of Educational Testing Service was 
an important factor in the success of the session. In addition to pro- 
viding meeting rooms and supplies, Anna Dragositz of El^S spent many hours 
of woi*k in making arrangements for the session. Her experience and work 
added a great deal to the success of the session. 

In general, I feel the meeting place used last year, the Continuing 
Education Center at the University of Chicago, was a better arrangement. 
Communication back and forth between Princeton and ETS was a greater 
problem than I expected and made evefning meetings very difficult. 




Uome participants folt the dciicription in the lilducational 
Researcher vrnri misleadinc;- The description of the course as beinc 
for applied researchers was relatively accurate, but more effort could 
have been made to indicate backgrpund necessary for multivariate 
statistics. One of the major problems we anticipated and tried to 
meet was that many researchers do not know what multivariate statistics 
are and are unprepared to find the topic difficult. 

The computer sessions were generally disappointing. The ETS 
computer staff apparently misinterpreted the directors' requests. 
Computer runs were not possible off hours or on weekends due to the 
ETS security procedures and the distance from the hotel. Computer 
runs were more limited than expected but by the last tvo days many 
participants were making runs successfully. The staff had prepared 
pre-run outputs which were distributed and discussed* The opportunity 
to submit computer runs was an important port of the objectives of 
the sessions and we had hoped it would be more successful. 

In conclusion, both staff and participants seemed very pleased 
with the' session. There was wide agreement that it served an important 
need. However , most participants felt it moved too fast or.d the 
preparation that they had was not adequate to fully benefit from - 
the session. This is probably related to the lack of courses 
such as this in most universities. Perhaps one of the bes.t measures 
of our success is that I have already received 21 requests for copies 
of the. handouts and 2^4 inquiries about next year, from persons who 
have spoken to participants in the session.. 
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Evaluation BY Participants. 

lif^E OF ^^P.^-^THM! MULTIVARIATE STATISTICAL ANALYSIS J 

Dl RECTOR • CHARLES E. WOODSON 

I A. To what extrAt did the relative availability or una vail abi lity of books 
and journals interfere with or promote your attempts to master the con- 
tent of this session? 

NONE 41% LITTLE -30% SOME 29% 



IB. To what extent did reproduced materials given to you by the staff improve 
matters? 

HLU^FUL 47$ 47% . VERY HELPFUL 53% 



2A. Did you feel that you lacked a "place to work", either alone or In smal 
groups? 

YES 11% NO 83% No COMMENT 6% 



2B. Was your room satisfactory? 

YES . 48% NO . 13% No COMMENT . 39% 



3A. Which features of the meeting rooms were inadequate or not conducive to 
learning? 

BLACKBOARDS .33% LACK OF OVERHEAD 

PROJECTOR ■ 1°^' 



ZE 


16% 


GHT 


25% 


R 


2SZ 



FURNISHINGS 0% 



. MISCELLANEOUS. 6% 



3B. Which features were especial ly f aci I i tati ve in the same regard? 
- LIGH T . u% ' ' • AI R ■ 13% 

SOUN D ..^i;/ FURNISHINGS 27% 

CONVENIENCE 39% No COMMEN T 33% 



/.A. fiis five days too long a period to leave your worK ax horr.e for The purpose 
of attending this session? 

YES 13% N O ' 87% 

4B. Was five days too short a period in which to learn much of the content of 
this session? 



YES 



63% NO 



5A. Were you allowed enough time in which to pursue activities of your own 
choos i ng? 

YES . N O •'-^'^ No COMy.EN T. 

5B. Would you have preferred not to meet in the evening after dinner? 

. YES 13% N O No COMMENT ^ 

5C. Would you have preferred more or fewer meetings per day than there actually 
were or was the number of meetings per day agreeable to you? 

FEWER 7% ENOUGH ' MORE 



6A. Wpre the individual lectures^too long to sit. and listen or take notes? 

YES .3% " ^-N NO 97^- 

6B. Were the lectures scheduled In an appropriate sequence? 

YES 31% NO 



7. ■ Did you have sufficient opportunities to interact with other participants? 
YES Riy . NO iqg' 



8A. Were the instructors too I naccessi b le or unapproachable so that you did not 
get the individual attention that you desired? 

■YES qy NO 77'/ . No COMMENT 14% 



88. Was it helpful to have graduate student assistants present? 

YES L'/. N O 4% No COMMENT .. 92% 



9A. Did the content of the lectures and readings presuppose rr^ore previous 
training than you had? 

YES 75% NO 25% 



98. Did the content of the lectures and readings presuppose • less previous 
training than you had? 

YES 8% NO 92% 



10. To what extent was the content of the lectures and readings relevant to 
what you hoped to accomplish during the session? ^ 

SOME ^2% MUCH ^^^^ No COMMENT 



MA. Were the lectures stimulating and interesting? 

YES(usually, or somewhat) • VERy""^^ 



IB, Were the lecturers competent to speak on the subject assigned thepf)? 
YES ^00^* NO " 



IC. Were the lecturers wel I prepared? 



YES 84,: ^0 



12. Were you disappointed in any way with the group of participants? 
YES 23% |yjQ 77%^ 



13. lf*you had it to do over again would you apply for this session which you 
have just completed? 

YES 8^'^ NO '^'^'^ 



14. If a session such as this is held again wouldyou recommend to. others like 
ybu that they attend?- ■ . 

YES NO "^^^ 



15. Do you anticipate maintai ni ng some sort of contact with at^least one of the 
sess ion staff? 



YES 61%^,,, N„ 39!!. 



Do you feel that AERA is making an inporrant conTriourion to educarion 
by sponsoring sessions such as this one? 

YES ■ 96% .NO A% 



Do you feel that anything* has happened during these five days to make it 
more likely that you will leave your present position of employment? 

YES 19% NO 81% 



Is it likely that you will collaborate in research .w i th someone else 
attending this session (other than those you already were likely to 
col laborate with)? 

YES 23% NO 77% 



Do you think that the staff should feel that it has accomplished its ob- 
jectives during this five-day session? 

YES 64% NO 36% 



American Educational Rcsavirdi Association 
Training Session on 

Mult:ivavicte Statit^t accI Anc^lysis In 
Educational Resciiirch 



at 

Educational Testing Service 
January 30 - February 3, 1971 
PARTICIPANTS 



John R. Andsrsoa 
Mid-Continent Regional 

Kducational L-iboratory 
10^ E. Indeiiendrncc Avenue 
Kansas City, llicsouri 64106 

Jenuy R. Armstrong 
University of Wisconsin 
A15 West Gilman Street 
Madison, V?isconsin 53706 

Daniel IBaer 

Department oi ?,«ychology 
Boston College 
Ches^.nut Hill 

Boston, Massachusetts 02167 

Paul F. Barbuto 
Teachers Colls^ie 
Cclumbia University 
525 H, 120th Street Box 37 
New York, New York 

Pdchard Beeson 

St. Louis University 

Researc]) Methodology 

221 North Grand Blvd. 

St. Louis, Missouri 63103 

Edwin W. Renter 

Unified School District ill 

13A1 Park Avo.nue 

Racine^ WisconR?aT 53403 

Dole E. Berger 
.DepartnKint of p5?ychology 
Ciarer.iont Cradu;ate School 
Clari-Ripnt. Cr^liiicrni^ 91711 



R. Darrell Bock 
Department, of Education 
University of Chicago 
5835 S. Kirabark Avenue 
Chicago, Illinois 60637 

Marvin W. Boss 
The Faculty of Education 
University of Ottawa 
Ottawa 2, Canada 

Paul Bradley 
UCLA 

Center for the Study of Evaluation 
145 Moore Hall 

Los Angeles, California 90324 

Robert L. Brennan 
Harvard University 

Graduate Sc:hool of Education - CAI Lab, 
.8 Prescott Street 
Cambridge, Massachusetts 02138 

Sara M. Brown 

Southern Connecticut State: Collcj;e 

501 Crescent Street 

New Haven, Connectictit 06515 

Alfred J. Butler 
Universi ty of Wi.sconsin 
415 West Gilman Street 
Madison,' VJisconain 53706 

y 

Robert ?. Cantrell 
Tennessee-* Re-ed Instittite 
3420 Riclinrds Street 
Nanlivilie, Te.nnossee 3721? 



MULTIVARIATE 



Sae Hyun Choe 
Fordha»n University 

4A Street 
Long Island City, New York 11101 

Robert W. Covert 
Teaching Associate 
Temple University 
Philadelphia, Pennsylavnia 

Fred L. Damarln 
Univetsity of Delaware 
22 A E. Woif Hall 
Nevmrk, Delaware 19711 

John de Jung 
University of Oregon 
Eugene, Oregon 97403 

Jean L. Dyer 

University of Louisville 
School of Education 
Belknap Campus 
Louisville, Kentucky 40208 

Normari Eagle 

Bronx Conmiunity College 

Bronx, Kew York 

R. Tony Eichelberger 
Dallas Independent School District 
' 3700 RoGS Avenue 
Dallas, Texas 75204 

Gerald L. Ericksen 
St. Olaf College 
Northfield, Minnesota 55057 

Robert A. Feldmesser 
Educational Testing Service 
Princeton, New Jersey 08540 

Jeremy D. Finn, Instructor 
Ontario Insl'ir.uto for Studies in Ed. 
252 Bloor Street West . 
Toronto 190, Ontario, Canada 

Patricia J. Vontes 
Salvo. Regina Coillej^ie 
33 Kay St?:o-t ^ 
Newport, Rhot'c Xsltind 



Garlie Forehand 
Carnegie-Mellon University 
Pittsburgh, Penna. 15213 

Gerald L. Frincke 
U.S. Naval Academy 
Stop 2C 

Annapolis, Maryland 21402 
Alan K. Gaynor 

The University Council for Ed. Adm. 
29 West Woodruff Avenue, Room 315 
Columbus, Ohio 43210 
. 

Leonard Giambra 
Miami University 
Oxford, Ohio 

, . Barry Greenberg 
>Iiami-Dade Junior College 
11011 SW 104 Street 
Miami, Florida 

Richard F. Haase 
University of Massachusetts 
243 VJhitemojre 

Amherst, Massachusetts 01002 

David L. Hayden 

NRRC/P Dept. of Special Ed. 

Box 911 

Harrlsburg , Penna. 17126 

George R. Holmes 
The William S. Hall 
, Psychiatric Institute 

Child & Adolescent Services 
Columbia, South Carolina 29202 

Maure Hurt, Jr. 

Arizona Center for Early Ch. Ed, 
College cf Education 
University of Arizona 
Tucson, Arizona 

Laird W. Heal 
Box 183 

George Poabody College 
Nashville, Tennessee 37?.u3 



>fJLTIVARXATE 



Paul j). Hood 

Far West Laborntory for Educational 

Research and Development 
1 Garden Circle 
Hotcil Claremont 
Berkeley, California 94705 

Nathan Jaspen 

New York University ^ 

100 Blcecker Street 

New York, New York 10012 

Virginia Keith 
The Faculty of Education 
University of Ottawa 
Ottava 2, Canada 

Ronald E. Lesher ' 

Bucks County Public Schools 
Court House Annex 

Doylesto\^, Penna. 18901 

Joel R. Levin 

Re^search & Developirent Center 
140 A Regent Street 
University of Wisconsin 
Madi.son, Wisconsin 53706 

Paul Van R, Miller 
Lehigh University 
School of Education 
Bethlehem/ Pennsylvania 18015 

■ ^ 

John II. Neel 
280 FAG 

University of Southern Florida 
Tan:pa> Flor4c[a 33601 

\ 

Tetsuo bkada ; 
U.S. Office of Education 
Washington, D. C. 20024 • 

Pietro J. PascalG^ 
Graduate Schooljof Education . 
Rutgers - The State University 
1-W Brunswicl:, K'iv. .'jersr.y 08903- 

|Timui:uy J. rcttlbonc 

I no:; -ij? . • 
j.as- uviKMvs, mGv; '"c*.:;ico 88001 



V/llllnm D. Pope.j'oy 
New York UnLverwity 
•157-07 Horace Harding Exprcsriwav 
Flushing, New York 11367 

Robert Pruzek ^ 
State University of New York 
1400* Washington Avenue 
Albany, New York 12203 

James Raths 

University of Maryland 

309 Dennis Avenue 

Silver Spring, Maryland 20901 " 

Buford Rhea 

Associate Professor of Sociology 
SUNY at Plattsburgh 
Plattsburgh, New York 12901 

... . 

William D. Rohwer, Jr. 
Institute of Human 'Learning 
University of California 
Berkeley, California 94720 

John Roscoe 

Kansas State University 
College of Education 
Manhattan, Kansas 66502 

Barbara Rossi 

State University at Buffalo 
Buffalo, New York 

David A. Sabatino 
Penn State University 
119 Rackley Building 
University Park, Penna, 16802 

James R. Sanders 
Indiana- University 
101 Education Building 
Bloomington, Indiana 4740.V 

Edwin Sause 

Research* Associate 

Kd, Res. Council of .Ancrica 

Rockefeller B-jllding 

Cleveland, Olilo ,44113 



inn-TIVA?NlATE 



Thomas Schwen . 
Ass't. Prof. Education 
Rm 10ft N Mitchell 
Indiana University 
Blooniinglon, Indiana 

Pio Scilligo 
Unrl varsity of Salesiana 
PZA Ateneo Salesiana 
Rome, Italy 00139 

Lowell Seymour 
Philadelphia Public Schools 
Philadelphia, Pennsylvania 

William Sharkan 
Bethlehem Art^:a School District 
3149 Chester Avenue 
Bethlehem; Pennsylvania 

Jack L. Shepler 

Indiana University of Pennsylvania 
.Indiana, Pennsylvania 15701 

George R. Shoemaker, Jr. 
Edinboro State College 
Edinboro, Pennsylvania 

Uichrtrd Shumway 

Department of Mathematics Education 
Ohio State University 
283 Arps Hall 
Columbus, Ohio A3210 

Ping Kee Siu 
Fordham University 
New York J, New York 

V. K. Small 
Curriculum Department 
Montgcmery Countzy Schools 
Rockville, liaryland 20805 

John C- Smart 
Assoc, Prof, of Education 
Glassbcro State College 
Glassboro, N. J. 08028 ^ 

R. I A. . Srvi.trU 

Un Jver:-! -I t*v_oj; Soulluirn Cal-! f orr»ia 

.^03 I* If r;,il.i !i:s H'lll 

Lor Ap-ivlcr;, CDMi:ora.ia 90007 



Gordon Taaffe 
Michigan Blue Shield 
Detroit, Michigan 

Neil H. Timm 

University of Pittsburgh 

A535 Forbes Avenue 

Pittsburgh, Pennsylvania 15213 

Raj^nond K. Tucker 
Department of Speech 
Bowling Green State University 
Bovling Green, Ohio A3A0A 

W. R. Unruh 
University of Calgary 
Calgary AA, Alberta, Canada 

/innie W. VJard 

Volusia Co. Schools 

801 - A North Wild Olive 

Daytona Beach, Florida 32018 

Jonathan R. V/arren 
Educational Testing Service 
Berkeley, California 9A70A 

G. Leigh ton Was em • 
Illinois State Department of Ed, 
50 Wedgewood Terrace 
Springfield, Illinois 62702 

William J. Webster 

Dallas Independent School District 

3700 Ross Avenue 

Dallas, Texas 7520A 

Norman Wexler 
Glassboro State College 
Glassboro, New Jersey 

Lawrence E. Wightman 
School of Education 
University of Massachusetts 
Amherst, Massachusetts 01002 

Arthur Uliite 

Ohio Slate University 

Arp/; Hrall (Science F.du. :atloo) 
High Street 
Col'iiii;!) Mf: , Ohio 43:?.l0 



; Asyocint.c IM'oFcsscu* of EducaUJon 
UnJvorslty of Miai:ii 
Coral Gables, Florida 3312/i 

Kor,er Woodbury 

N.C, Roai-d of Juvenile Correction 
900 Doi^\>7ood tane 
Raluigh, North Carolina 27607 

M. I. Chas. r,. Woodson 
4641 Tolm.'in Hall 
llniv»?.rsity o£ Calit'oTnin 
15er:<eley, California 94720 



Allen Yaiics 
V ti y r.l I o 1 o i;, y Do par t: x c n f 
liov;lini:|. Groun I'nivorslty 
Bowling Green > 01i\o 43m03 

John F. Zimnier 

Minnesota StaLc College Roard 
550 Cedar Street 
St:. Paul, Minne.^iota 

Barry ZimmeiTnn 
Universit.y of Arizona 
8535 E. Ikllcvue 
Tucson, Arizona 85715 
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Research Training Session 
The Research Component of Black Studies 
LaMar P. Miller, New...York University 
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Title: The Research Component 
Staff: 

Dr. LaMar P. Miller 
(Director) 

Dr. Roscoe C. Brown Jr. 

Mr. Ewa Eko 
Dr. Edgar G. Epps 
Dr. Edmund Gordon 
Dr. Henry Simmons 
Dr . Francis Botchway 

Mr. James Elsberry 
Dr. James Rosser 



of Black Studies 



Educational Research Director 
Institute of Afro-American 
Affairs 

New York University 
Director 

Institute of Afro-American 

Affairs _j 

New York University 

Coordinator 

Six Institutions Consortium 
Bennett College 

Professor of Education 
School of Education 
University of Chicago 

Chairman, Guidance Department 
Teachers ' College 
Columbia University 

Chairman 

BlacH Studies Department 
University of Indiana 

Director 

Institute of Afro-American 
Studies 

Richmond College 

Director of Community 

Department 

Research Division 

Center for Urban Education 

Office of Academic Affairs 
University of Kansas 
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Dr. Ronald Walters Director 

Black Studies Department 
Brandies University 

3. General Description 

One of the most publicized and controversial field in education 
over the past two years has been black studies. Although there is 
great concern about the conduct of black studies, universities, 
secondary and elementary schools continue to establish programs. 
Moreover, there is little agreement among scholars or students on 
how to define the field, who should develop it, what research is 
needed and how the research should be conducted. 

From the onset of black student struggles for black studies 
programs, generated primarily on college campuses through the recent 
period of increased demands for black studies on elementary and 
secondary levels, educational institutions have faced new questions 
and troubling issues. If research is to be of assistance in helping 
schools to objectively include the contributions of various ethnic 
groups in the c)irriculum- we need to be able to delineate whe^t. 
factors are currently operating in the system so that we can 
encourage or discourage choices. While many shortcomings and 
advantages in the curriculum are obvious to scholars, school men 
and Black Americans, there are specific questions that have not 
been raised and problem areas left not well understood. The 
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objectives of the training session provided direction for those 
seeking to improve the field as an academic discipline through 
research. " - 

A frequent criticism of black studies is that no adequate 
foundation has been developed upon which to base instructional 
programs. Black scholars note that in historical, political, 
economic, psychological and sociological research one can find 
examples of gross negligence and cite instances of distortions 

or half-trutns in allegedly respectable work concerning black 

...j 

people. If we consider a field in which the systematically- 
synthesized knowledge may be. either sparse or distorted, then the 
principle component of any ethnic or Afro-American studies program 
ought to. be concerned with expanding the discipline base upon 
which the studies are built. f" 
4. Objectives : 

The research training session was designed to provide methodo- 
logical and non-methodological information for those who are 
involved in developing and evaluating programs or working in related 
research areas. The program, aimed to help participants acquire 
an understanding of the nature and proper place of black studies in 
education so that they would be better^ equipped to develop lasting 
programs in various settings. The three primary aims impleme.nting 
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the program purpose were development of participants knowledge 
regarding basic ideology, organization and structure and program 
content. The objectives of the program were to: 

1. examine existing research methodologies which can- 
aid in establishing a solid discipline base for black 
studies. 

2. describe new and innovative methods of research which 
seem most appropriate from a black scholar's point of 
view. 

3. outline and encourage new areas of research that are 
related to black studies. 

4. report on research projects involving black studies 
that are currently in progress or that have recently 
been completed. 

5. examine new approaches to curriculum development in 
black studies and related areas. 



5. Schedule: 

First Day: Session 1: 

Orientatibh: "statement of objectives, introduction 
of staff with identification of their special 
competencies and interests, group discussion of 
participants, specific interests and needs. 

Session 2: 

The Relationship of the * Community to Research in 
Black Studies. A Summary of complaints of black 
students on current educational programs. 
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Second Day: 



Session 1: 

African Cosmological Ideas: Methodological 
Approaches. A delineation of African 
cosmological philosophy as the basis for 
the African component of alack studies programs 



Third Day: 



Fourth Day: 



Session 2: 

Research/ Curriculum Development and L^lack 
Studies in Schools of Education and in 
Elementary and Secondary Schools. 

Session 1: 

Problems frequently overlooked that pertain 
to research in the Black community. 
Research as a component of Black studies 
programs . 

Session 2: 

Research/ Curriculum Development and Black 
Studies in the University. 

Session 1: 

Some Research Designs for Evaluating Black 
Studies Programs . 

New Strategies in Educational planning and 
Research for Black Studies. 



Session 2 : 

Methodologies of Research on Achievement 
Orientation of Black Consciousness. 

Evening Lecture: 

A Saga of Black History 

Alex Haley Author of the Autobiography of Malcolm X 



Fifth Day: 



Session 1: . 

The Methodology of Oral History in Expanding 
the Discipline of Black Studies. 



Session 2 : 

An Analysis of the Performance of Black Students 
in a Black Studies Program.. 

New Directions for Black Studies as a Discipline. 



6. Participants: 

The age of the twenty participants ranged from 27 to 53. 
Fourteen of the participants were under 40 and three were under 
thirty. Six. of the participants were female^ Only four of the 
participants were members of AERA and none had attended previous 
AERA pre-sessions. 

Half of the participants held the doctorate and four were 
candidates for the doctorate. One participant held a lav/ degree* 
and the others held M.A. degrees. 

Eighteen of the participants were from a college or university. 
One participant worked for a research consulting firm and the other 
was part-time lawywer and part-time Director of Black Studies Program. 
All of the participants, held positions that were in some way connected 
with or related to the development or evaluation of Black Studies 
Programs . 

The following list is a break-down of the positions held by the 
participants: 

Coordinator Ethnic and Cultural Studies 

Director of Research 

Director of Black Studies 

Coordinator of Urban Studies Program and 

Assistant Director of Program to Develop . 

Social Sttidies Units 
» Program Analyst and Designer, of Mental Health 

Programs - Consultant to Special Programs 

for Disadvantaged 
Teacher - graduate courses - Introduction to 

Research, 1 Under Graduate Course - Educational 

Psychology 



Professor of History and^ Chairman of Social 

Studies ^^.^^^ X 

Professor of Curriculiim and Research on — ^ 

Problems of Indian Pupils 
Chairman Black Studies 

Assistant Professor - Higher Education 
Research Assistant- Coordinator of Career 

Opportunities Programs 
Director Black Education Center - Professor 

of Curriculum 
Graduate Students 

Director - Center for Afro-American Studies 
Participants represented ten states; New York, Tennessee, 
California, Florida, Ohio, Nebraska, Maryland, Illinois, 
Michigan and Indiana. One student was from Toronto, Canada* 



?• Instructional and Evaluation Materials: 

Instructional reference materials were distributed during 
the pre-session. These consisted of recent papers by ataff 
members, bibliographies, directories of Afro-American and Afro 
Resources, bibliographies of multi-media resources on Afro- 
American Studies and a Directory of Afro-American Studies and 
Departments in various colleges and univeirsities around the 
country. All of the resources of the Institute of Afro-American 
Affairs at New York University were made available as well as the 
Schbmberg Library in Harlem* The Institute of Afro-American 
Affairs donated copies of their materials such as the literary 
magazine. Black Creations , Dr. Miller's report on Black Studies 
In The 70' s, and copies of the New York University Educational 
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Quarterly on Black Studies , Since many of the participants were 
actively involved in the development of programs they contributed 
by sharing their information on instructional and evaluational 
materials. Texts of a general nature, were also provided such as 
Teaching the Black Experience (Banks) , and B lack Studies in the 
University (Robinson) . 

8. Evaluation: 

Verbal feedback was illicited in informal r individual and 
small group conferences throughout the pre-session. Written 
observation or critiques were submitted by the participants 



1;.;; following the pre-seesion. In general, participants judged the 

program as a highly useful experience that provided valuable 

P' 

information. Since one of the major objectives was to provide 

direction for participants an effort will be made in the form 



Ml of ^ follow-up questionnaire. In addition to this, the director 



of the pre-session, has maintained close contact with 70% of the 
participants. Most have been back to the Institute of Afro- 
American Affairs for additional assistance and consultation. 
There were, of course, some negative aspects of the pre-session. 
The physical facilities were inadequate. While regular staff 
members were to hold individual conferences with participants^ 

thbsewho came in as part-time instructors did . not have this 



opportunity. :| 
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The participants were most interested in having a pre-session 
such as this one repeated. While a great deal of ground was 
covered^ they felt that we had indeed only scratched the 
surface. On the other hand, they were particularly pleased 
that the offerings were specific and to the point, 

9. Directors Evaluation: 

In general both the staff and the participants felt that the 
pre-session was successful. Probably the best single index of 
this success was that the participants overwhelming indicated 
that the pre-session should be offered again. 

What we were able to do in this pre-session was but a small 
segment of the story of the developing discipline of Black Studies^ 
Of primary value is that hopefully this experience will help 
delineate some of the more serious issues of Black Studies, 

While many of the sessions delt with issues that have been debated 
at Black Studies Conferences, there was far more specificity 
concerning ideology, but participants and speakers clearly recognize 
that the organization and such things as scope and sequence of 
programs depended on the philosophy of those conducting or planning 
programs. 

In this pre-session, unlike others I have attended, the 
backgrounds and interests of the participants were quite similar. 
Their expectations, however, varied. Perhaps we could hjave done 
more to have a better indication of what, their expectations were, 
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This kind of pre-session, in an area that is in itself contro- 
versial/ should be and was quite different than those dealing 
with strictly methodological concerns* In many ways^ if it is 
true that it was valuable^ it was also relevant. The outstanding 
qualifications of staff members in a pre-session such as this one 
is highly unusual. I ihink it would be difficult to bring 
together this kind of expertise in any other geographic area 
except New York City. | 

In all probability the 'most important function of the pre- 
session was thr'. it clearly suggested that the future development 
of Black Studies must be based on research. We do not know the 
extent to which the pre-session encouraged future research. This 
aspect^ of course, is difficult to evaluate at the present time. 
We did, however, raise some critical issues for research in an 
attempt to stimulate those interested in improving Black Studies. 
We also believe we were ab],e to define the perimeters and 
dimensions of a new and improved field. 
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P&A 1971 tesEARCH Training Sessions 

EVALJATION BY PaRTIC1PA.\TS 



.^Jame of Session:. "^"^ "^'"^'-^ component of black stl-dies 
Director:. . lamar miller — ~ 



lA. To what extent did ihe relative avcii lab i I it/ or unavailability of books 
end journals interfere with or pro'note ycur attempts to master the con- 
tent of this session? 

' NONE LITTLE ^8% gOME 



IB. To what extent did reproduced materials given to you by the staff Improve 
matters? 

HELPFUL 60% VERY HELPFUL 40% 



2A. Did you feel that you lackad a "place to work", either alone or in small 
groups? 

YES 11^ NO '72% No COMMENT 17% 



2B. Was^^ur room satisfactory? 

YES 59% NO 35 No COMMENT . 6 



3A. Which features of the meeting rooms were inadequate or not conducive to 
learning? 



BLACKBOARDS • 33% LACK OF OVERHEAD 

PROJECTO R 11% 

S I ZE . ' 0 







FURNISHINGS 


0 


LIGHT 


10% 










MISCELLANEOUS^; 


5% 


AIR 


72% 







3B. Which features were especially facilitative in the same regard? 

LIGHT 1 1% AI R 0 

SOUN D. 33% FURNISHINGS 11% 
CONVENIENCE 22% No COMMENT ^3% 
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<^»A. Was five days too long a period to leave your woc*^ at home for the purpose 
of attending this session? 

YES 12%' • NO 83% 



4B. Was five <1ays too short a period in which to learn much of the content of 
this session? 

YES 12% "NO 88% . 



5A. Were you allowed enough time in which to pursue activities of your own 
choosing? 

YES 86% ' NO 7% No COMMENT 7% 



53. Would you have preferred not to meet In the evening after dinner? 

YES 17% NO :>&% No COMMENT .-^^7% 



50. Would you have preferred more or fewer meetings per day than there actually 
were or was the number of meetings per day agreeable to you? 

FEWER 12% ENOUGH 88% MORE 0 



6A. Were the individual lectures too long to sit and listen or take notes? 

YES 6% ■ N O 94% 

6B. Were the lectures scheduled in*an appropriate sequence? 

YES 82% NO 18%. 



?• Did you have sufficient op:?ortuni ties^ to interact with other participants? 
YES .100% NO 0 



8A. Were the instructors too inaccessible or unapproachable so that you did not 
get the individual attention that you djesired? 

^ * YES * 6^ NO 76%. . No COMMENT 18% 



8B. Was it helpful to have qraduate student assistants present? 

YES 65% NO 0 No COMMENT 35% 
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9A. Did the CG^itent of the lectures and readings presuppose more previous 
training than you had? 

YES jj^ NO ^ ^ 

98. Did the content' of the lectures and readings presuppose less previous 
training than you had? 

28% ' . • 72% 

YES NO ^ 



10. To what extent was the content of the lectures and readings relevant to 
what you hoped to accorpplish during the session? 

SOME 33% MUCH 61'^ No COMMENT 6% 



MA. Were the lectures stimulating and interestipg? 

* YES(usua*l ly, or somewhat) VERY 

I IB. Were the lecturers competent to speak on the subject assigned- them? 
YES NO 



lie. Were the lecturers well prepared? 



YES NO 



12. Were you disappointed In any way with the group of participants? 

YES NO 88% 

13. If you had i t to do over again would you apply for this session which you 
have Just completed? 

YES 94% NO 6% 



14. If a session such as this is held again would you recommend to others like 
you that they attend? / 

YES J8% NO 12% 

15. Do you anticipate maintainina. soids sort of contact with at least one of the 
session staff? 



YES 



^ . 68? no' 12% 



16. Do you feel that AERA is making an important contribution to education 
by sponsoring sessions such as this one? 



YES 94% NO 6%^ 



17. Do you feel that anything has happened during these five days to make it 
more likely that you will leave your present posi;t"ion of employm^int? 

YES 0 NO 100% 



18. Is it likely that you wij.l collaborate in research with someone else 
attending this session (other than those you already were likely to 
col I aborate wi th) ? 

YES A4% NO 56% ' 



19. Do you think that the staff should feel that it has accomplished Its ob- 
jectives during this five-dav session? 

YES 94%' NO • 6% 
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AERA 1971 Research Training Session 



The Research Component of Black Studies 



PARTICIPANT ROSTER 



1. Dr. John P. Bailey, Jr. 
38 Brentwood Avenue 
Jeiicksonville, Florida 

2. Dr. Margery Butler 
State University College 
New PalLz, New York 

3. Mr. Milfred Fierce 
Vassar College 
Poughkeepsie, New York 

4. Dr. Chester W. Gregory 
Coppin State College 
2500 West North Avenue 
Baltimore, Maryland 

5. Dr. Ephraim Issaac 
Harvard University 
77 Dunster Street 
Cambridge, Mass. 

6. Mr. Harold W. Hoton 
Ctr. for V&T Ed. 
Ohio State University 
1900 Kenny Road 
Columbus, Ohio 

7. .Mrs. Margaret Hardisou 

108-14 Ditmars Blvd. 
East Elmhurst, New York 

8. Mrs. Stephanie Howard 
Coppin State College 
2500 West North Avenue 
Baltimore, Maryland 

9. Mr. Amos Isaac 
1328 Calhoun 
Redlands, Calif. 



10. Miss Edwina Johnson 
Fordham University 
113 W 60 Street 
924 B 

New York, New York 

11. Mr. Fred Kimbrough 
20 North Grand Blvd. 
St. Louis, Missouri 

12. Dr. Carleton Lee 
Western Michigan University 
Kalamazoo, Michigan 

13. Dr. John Petry 
Bureau of Educational 

Res. & Services Ball 302 
Memphis State University 
Memphis, Tennessee 

14. Miss Grace Porter 
1325 Suburban Apts. 
DeKalb, 111. 

15. Mrs. 0. Jeane Ramey 
Vassar College 
Poughkeepsie, New York 

16. Miss Tanya Russell 
1 Garden Circle 
Berkeley, Calif. 

17 . Dr . Charles Sherman 
Dept. of Ed.- 

Illinois STate University 
Normal, 111. 

18. Miss Carrie Smith 
575 Herkimer Street 
Apt. 7B 

Brooklyn, New Y^*:-- 
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19. Mr. Melvin Wade 
Black Studies Program 
Box 688 

University of Nebraska 
Omaha, Nebraska 

20. Mr. Joseph Young 

Institute of Urban Education 
Room 648 

55 West 42 Street 
New York, New York 



Long Term Follow Up 
of 

1971 AERA Research Training Sessions 

r 

Since 1966 AERA has sponsored research training activities. Some of 
these training activities have been workshops preliminary to the regular 
Annual Convention. A regular part of the "presessions" has been their 
evaluation. Each presession director has planned and conducted some sort 
of evaluation of his presession, usually on the final day of the workshop. 
In some cases these evaluations hsve been exams covering the skills and 
concepts taught, and in other cases they have been more in the vein of 
attitude surveys. In recent years the Association has conducted its own 
post-training session evaluation. These have been primarily designed to 
obtain information from participants relative to the relevance, effective- 
ness and utility of the presession topic and instructional procedures as 
well as the logistical support of accomodations and services available to 
the trainees. 

Prior to the 1971 Research Training Sessions there had been no long 
term follow up on the possible effects of the sessions on the research and 
research related behaviors such as teaching, administratiot. and service. 
Clearly, if the Association is going to continue its support of these 
activities, there should be some evidence of the sessions' influence on 
what the trainees do when they return to their home institutions. 

Aa a first attempt to obtain such data, a survey instrument was 
devised which solicited information from those attending the 1971 Research 
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Training Sessions. Often such questionnaires ask the respondent if he 
thinks the experience he has had (the training) has made a difference in 
his attitude or feelings i In the instrument developed to assay the 
effects of the 1971 presessions, the questions were worded in such a 
way as to inquire into the actual research related behaviors. For 
example, respondents were asked three questions relating to the frequency 
of their communication with their colleagues on topics relating to their 
presession. They were not asked if they felt their communication about 
research ideas related to the presession had changed. 

The questionnaire had two major parts. The first was general and 
was directed to all trainees irrespective of the particular session they 
attended. In addition,' a second part contained questions related to 
specific presessions. Each director was contacted and invited to submit 
a series of items relating specifically to the use being made of the 
concept and skills taught in his presession. Three of the nine directors 
responded with a set of questions. They were included as the second 
part of the questionnaire sent to their "students". 

Each respondent was asked to provide information on his general 
research activity since the presession, the identification of his 
presession, and more specific information if the presession he had 
attended was one of the three mentioned above. The presession 
identification would permit analysis of the returns by presession. 
Mailing lists were prepared from the enrollment records in the AERA 
Central Office. The first mailing went out about November, 1971, 
approximately nine months after the 1971 presessions were held. There 
was a follow up letter sent two weeks after the first mailing to remind 
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the respondents to complete and return the survey. Appendix A presents 
the general part of the final survey instrument. 
Results and Analysis 

-S' 'I 

The data were collected and coded for analysis in the AERA Central 



During a preliminary analysis of the data, certain inconsistencies in the 
identification of the presession led us to make a thorough check of the 
mailing lists. This check revealed that an error had occurred in their 
preparation which resulted in a failure to send questionnaires to all 
those who did attend a presession. The nature of this error aud its 
consequences for subsequent analysis can be seen in Table 1. 



I- 



Office in Washington. Final returns and code sheets were sent to Michigan 
State University for key punching and analysis about January 15, 1972. 





Table 1 



Enrollment, Sanpling Proportion 
and Response Rate for Each 1971 Research 
Training Session 



Presession Number Proportion Response 

Director (s) Enrolled Sampled Rate 



Miller 


20 


1.60 (32)* 


.25 (8) 


Woodson 


77 


1.09 (84) 


.57 (48) 


Farley 


62 


.81 (50) 


.48 (24) 


Ryan 


22 


1.00 (22) 


.59 (13) 


Baker & Popham 


91 


.38 (35) 


.94 (33) 


McSweeney & Porter 


48 


1.00 (48) 


.48 (23) 


Tracz & Bnmo 


48 


.94 (45) 


.49 (22) 


Rothkopf & Frase 


38. 


.63 (24) 


.50 (12) 


Totals 


406 


.84(340) 


.54(183) 



*See text for meaning of sampling proportions in excess of 1.00. Numbers 
in par an theses indicate frequencies. 
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As can been seen, some of the presessions were badly undersampled and two 
were oversampled. This latter occurred because some individuals attending 
one of the presessions were erroneously included in Miller's and Woodson's 
sessions. We have no reason to believe that we sampled individuals not 
attending any presession. Since this error was not discovered until long 
after the original mailing and even the follow up mailing, we had to live 
with it. The consequence was that no comparison among presessions could 
be made. All analyses had to be restricted to the total group. 
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Table 2 presents the percent, mean and standard deviation for the 
responses to items one to ten and fourteen of the general questionnaire. 
The majority of respondents had not experienced any increase in the number 
of presession-related letters they have exchanged, nor had they noted an 
increase in the number of requests for articles and materials related to 
the presession topics; The majority did report, however, that they had 
req.iested more articles and materials on the presession in the period 
following the end of the presessions. 

The next two questions dealt with independent and collaborative 
research activity. In both cases the majority of the respondeits 
reported an increase in activities. Over 75% of the respondents reported 
an increase in the time they devoted to reading aj;d thinking about the 
research related to the topics considered in the training session they 
attended. 

Questions #8 and #9 solicited information on changes in the skill 
related to planning and designing of research activities. Again, over 
75% of the respondents reported an increase in their ability to see 
qualifications and complicating factors in the research reports they had 
read since completing the presession. They also rc-ported an increase in 
their skill in planning and developing research activities. 

It was believed Important to ascertain how research training sessions 
might influence teaching activities. Question #10 was developed to 
obtain data relevant to this problem. A sizeable majority of the respon- 
dents reported an increase in their recognition of the relationship 
between instruction and the presession topic. Finally,^ a majority 
indicated that ,the presession did not change their involvement in 
professional activity. 

m 



8. 



Questions #12 and tfl3 sought to determine the degree and nature of 
the "student's" efforts at continuing his learning about the topics 
related to the presession. Eighty-three percent said they had made an 
attempt during the nine months following the presession to add to their 
understanding of the problems discussed during their training period. Of 
those reporting continuing involvement in learning, 15% said they accom- 
plished their study by reference to materials handed out during the 
presession, 22% reported using textbooks, 44% reported using a combination 
of presession materials and textbooks, and the remaining 19% reported 
using short-courses and various combinations of the two most preferred 
means. 

Another way in which the presessions might have had an influence 
on their students is through their (students) allocation of time. Question 
#15 sought information on this point. Table 3 presents the results of 
this question. 
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Table 3 



Response to Question #15 
Percent of Time Allocated to Various 
Professional Activities Before and 
After the Presession 

Percent of Mid Category Teaching Research Administration Service 

Time Devoted Score Before After Before After Before After Before After 



0 to 20 


10 


48.21 


49.40 


40.48 


35.12 


66.67 


66.07 


80.36 


83.33 


21 to 40 


30 


13.69 


14.88 


20.83 


27.98 


14.29 


13.10 


12.50 


10.71 


41 to 60 


50 


26.19 


21.43 


17.26 


14.88 


8.93 


10.71 


5.36 


4.17 


61 to 80 


70 


6.55 


9.52 


8.33 


8.33 


4.76 


5.95 


1.19 


1.79 


81 to 100 


90 


5.36 


4.76 


13.18 


13.69 


5.36 


4.17 


.60 


0.00 


Mean 




31.4 


31.4 


37.1 


37.6 


23.4 


23.9 


19.0 


15.0 


N 




168 


168 


168 


168 


168 


168 


168 


168 



Table 3 was prepared by first categorizing the percents recorded by 
the respondents into one of five categories; 0 to 20%, 21 to 40%, 41 
to 60%, 61 to 80% and 81 to 100%. The mean tines reported on the 
bottom of the table were computed by using the mid-category (10, 30, 
50, 70 and 90) as the score for that category. As can be seen from 
Table 3, there is very little, if any, evidence for a change in the 
allocation of time across the four activities of teaching, research, 
administration and service from before the presessions to after. 

Questions m and //l6 dealt with the incidence of scholarly and 
teaching activity related to the presession. Over two-thirds (66.86%) 
of the respondents reported they had written a scholarly paper which 
they believed directly benefited from their presession attendance. 
However, , only 25% indicated they had planned courses that were related 
to the presession topics. 
Factor A nalysis of Qtiestionnalra 

Since this was the first attempt at a long term follow up of the 
effects of the AERA Research Training Sessions, and there might be others 
in succeeding years, it was decided to determine the factor structure 
of the items included in the survey instrument. Items one through twelve, 
fourteen and sixteen (see Appendix A) were included in the analysis. 
There were useable responses from 181 respondents for the first stage of 
the analysis. This first stage was__a principal axis analysis of the 
matrix of intercorrelations (TablL 4). ) The first four roots had Eigen 
Values greater than one (4.92, l.i/riCiO, and 1.03). 
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12. 

These roots accounted for approximately 60% of the variance. The Eigen 
vectors associated with the first four roots were rotated to "simple 
structure" by the varimax criterion with the results as portrayed in 
Table 5. 

Factor I is characterized by high (greater than .50) positive loadings 
on the following items: time devoted to reading the literature (//6), time 
devoted to thinking about research problems 07) , tendency to see qualifica- 
tions and complications in the literature (98) , skill in developing and 
planning research activities (#9), seeing of relationship between the 
literature and instruction (#10), and making attempts to increase knowledge 
of literature 012) . The highest two loadings were on items dealing wit:h 
-the commitment of time (#6 and 97). Since the remaining high loadings, 
though not treating time directly, do indicate an allocation of time to 
research activities related to the presesslon topic; the factor might be 
characterized as a motivational factor tapping the depth of the individual's 
involvement and commitment to the research area covered by the presesslon 
he attended. 
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Table H 

Varimax Rotatloral Analysis 
Questionnaire Items 
Numbers 1-12^ 14 6e 16 



Variables 
(Item No.) 



Factor I 



Factor II 



Factor III 



Factor IV 



1 


.0817 


.8152* 


.1848 


.0715 


.7105 


2 


.1825 


.7699* 


.2119 


.0455 _ 


.6731 


3 


.4011 


.58?8* 


-.0286 


-.0739 


.5068 


4 


.1175 


.5603 


.5720* 


.1374 


.6738 


s 


• 2129 


o o o o 

. 3233 


.6660* 


.0070 


.5935 


6 


.7407* 


.3222 


.0032 


-.1312 


.6696 


7 


77QQWU 




.1897 


.0262 


.6914 


8 


.6162* 


.1151 


.1600 


.2847 


.4997 


9 


.5848* 


.1762 


.2329 


,3369 


.5407 


10 


.6682* 


.0825 


.2698 


.2720 


.6001 


11 


.1415 


-.0090 


.7955* 


.0489 


.6554 


12 


.5504* 


. -.0265 


.3270 


-.3080 


.5054 


14 


. 1289 


.0126 


.0709 


.8656* 


.7710 


16 


.1651 


.1417 


.6169* 


.0610 


.43i7 


;h Loading 


.7788 


.8152 


.7955 


.8656 




•portion of Variance 


.2107 


.1600 


.1546 


.0834 





Cumulative Proportion of 
Variance 



.2107 



,3707 



.5253 



.6088 
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Factor II had only three Items with loadings in excess of .50. All 
three of these items dealt with rather overt communication behaviors; 
i.e., letters exchanged (#1), requests for articles and/or materials (#2) 
and requests sent out for articles and/or matWrials (#3) . 

Factor III had four items loading on it. They were, in order of 
size: publication of a paper which benefited from the presession (#11), 
planning of courses related to the presession topic (#16), independent 
research related to presession (#5) and presesoion related collaborative 
research (#4). Where Factor I seoned to be related to. motivation or 
commitment to research. Factor III seems to be more concerned with the 
product of research conmiitment or the mamier ^^^^^^ which that product is 
generaited. 

The final factor, IV, had only one item with a loading in excess of 
.50, number 14, dealing with involvement in professional associations since 
the presf%9sion. 
Simmxary 

A survey of the long term (nine months) effects of the research training 
sessions sponsored by AERA prior to the 1971 Annxial Convention was conducted. 
An instrument was designed to obtain data from those Association members 
attending the presessions relating l:o their presession related research 
activities. The results showed some rather large, self-reported effects of 
those responding. A factor analysis of the questionnaire Iteas was reported 
which yielded four factors. 
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Appendix A | 



The general questions, to be answered by all respondents , refer to your behayior or 
m activities since your attendance at the presession. Please circle your response to each 
% question. 



J. The time that I devote to thinking about problems related to the Presession topic has, 

remained 

^ decreased the same increased 

ii 1 2 3 4 5 6 7 

I 8, My tendency to see qualifications and: complicating factors in the literature that I 
^- have read recently (within 9 months y r^^^ 

.-remained 

fj, decreased the same increased 

1 2 3 4 5 ,6' 7 

I 9. My skill in developing and planning research activities related to the Presession 
topic hae, 

remained 

decreased ; the same increased 
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1. The number of letters that I have exchanged related to the Presession topic has, I 

remained . . " 1 

decreased the same increased 1 

I 1 2 3 4 5 6 7 i 

m. ' .: . . ! 

2. The number of requests that I have received for articles and/or materials related j 
to the Presession topic has, 

remained 

decreased the same increased I 

I 1 2 3 4 5 6 7 : 

3. The number of requests that I have sent out for. articles and /or materials related 
to the Presession topic has, 1 

remained ; 
decreased the same , increased | 

I • 1 2 3 4 . ' 5 6 7 ' 

• k: ■ . • ■ r . , . ■■ . ; 

I 4, My collaboration with others 6^ related tp the PreiBession t^^^ has, 

I ' ■; , ; ■ ' " ' : reioiained. '. 

I decreased the a^ame increased 

1 2 3 4 5 6 7 . 

I 5. My independent research related to the Presession topic has, 

remained 

^ decreased the same increased 

1 2 3 4 5 6 7 

I ^ . . • . ... . . ■ . ", . ■ ■ ■ • .■ ■ • - ■ . ... 

I 6, The time that I have devoted to reading the literature related to the Presession 
topic has, 

remained 

decreased the same increased 

1 2 3 4 5 6 7 



16. 



10* The relationships that I see between instruction (teaching/research) arid the topic 
of the Presessiori has, 

remained 

decreased thin same increased 

12 3 4 5 6 7 

11. Have you written a scholarly paper either published or unpublished which ycu feel 
directly benefited from your attendance at the 1971 Presession? 

No Yes 



12. Have you made an attempt in, the last 9 months to increase your knowledge of (read 
here, the topic of the Presession you attended) as a result of your attendance at 
the 1971 Presessions . (i.e. , did you do something you might not have done if you 
had not attended)? 

No Yes 

13. If you answered the above question "yes" indicate how by circling^^^one or more of 
the following alternatives: 

a. By studying the instructional materials handed put at the Presession. 

b. By independent study from textbooks. 

c. By attending a "short-course" or •'workshop" on the sane or/a clearly related 
topic* 

d. other (please specify) : _^ 



14. Since, the 1971 Presessions my activity ii^ professional associations has, 

remained 

decreased the ^ame increased 

1 2 3 4 5 6 7 

15. Indicate the approximate distribution of your time prior to and after the 1971 
Presession. 

Prior to Presession After Presession (e.g., 9/71) 

Teaching ^ ■ ' 

Research ■ 

Administration ■ 

Service 



16. Have you planned and/or developed any new courses related to the Presession topic? 

No • Yes 
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IfFICIAL NEWSLETTER OF THE AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 



Volume XXI, SPECIAL EDITION, Nov. 1970 



If?' 



This special edition of ihc Educational Researcher was instituted in order to transmit the enclosed information to 
the membership in the most expedient manner. Pages I through 4 eontain information on and an application for 
AERAV Research Training Ses^ons to be held prior to the 1971 Annual Meeting; pages 5 through 16 contain 
the proposed new Association Bylaws which require membership ratification. 



1971 Research Training Program 

GENERAL INFORMATION PROGRAM DESCRIPTION 



APPLICATION 



AN APPLICATION FOR PARTIAL FUNDING has been submitted to the United States Office of Educa- 
tion to enable AERA to conduct for the fifth year a program of intensive training sessions on research tech- 
niques and methodologies. Eight five-day sessions will be held January 30-February 3, 1971, just prior to 
the Annual Meeting, February 4-7. Seven of the sessions will be held in New York City; the eighth will be 
held in Princeton, New Jersey. This year's request for partial rather than full funding of the Research 
Training Program results from recent cutbacks in federal funding of education. AERA must initiate par- 
tially .self-su§taining Research Training Sessions to insure the continuation of the program/A registration 
fee of $50.00 will be charged to allay session costs not covered by the federal grgnt.* 

The AERA Research Training Sessions are carefully selected, designed, and organized for various 
audiences of full-time research producers from the most sophisticated to those whose original graduate 
training contained only minimal research preparation. The rapid growth of methodological and technical 
skills being employed in educational research is such that every researcher must continue to receive train- 
ing, not only in old established methods but also in new methods that are being proposed. The Research 
Training Sessions are expressly designed to meet these needs. 

Participation in the Research Training Sessions is not restricted to members. Applications will be 
processed in the order they are received. Those who plan to attend are urged to apply early as all sessions 
will have limited-enrollment. (Please note, however, that a session will not be held without a minimum 
enrollment of 20 participants.) January 15, 1971 is the deadline for. the submission of applications. Most 
applicants may expect to be notified of the decision of the director of the session for which they have 
applied within three weeks after the receipt of the application. UPON NOTIFICATION OF ACCEPTANCE, 
please submit the registration fee to the AERA Central Office as soon as possible since registration will not 
be complete until this fee is paid. In the event of cancellations, full refund will be made if notification 
is given to the Central Office before January 22, 1970. From this date until January 29, a service charge of 
$10.00 will be assessed; no refunds will be given after that date. 

*An additional $2*5.00 for cost of materials vvi II be charged for the session conducted by Charles 
Woodson. 
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1. THE RESEARCH COMPONENT OF BLACK STUDIES 

Director: LaMar P. Miller, Education Research Direclor. In- 
slilule of Afro-American Affairs. New York Uni- 
versity, Washington Square Education Buildings 
Room 778, New York. N.Y. 10003 

Staff: Roscoc C. Brown. Jr.. Institute of Afro-American 
AfTuirs; Ewa Eko. Bennett College; Edgar G. 
Epps. Tuskegee Institute; Edmund Gordon. Co- 
lumbia University; Henry Simmons. Univcraliy of 



Indiana; Francis Uotchway. Richmond College: 
James Elsbcrry. Center for Urban Education; 
Sylvia Obradovic. Far Wcsl Laboratory for Edu« 
cational Research and Development; James Ros> 
ser, Southern Illinois University 

Conknt and Objectives: The most publicized controversial field 
over ihc past two years has been black studies. The ohjcciivcs 
of this prcsc5sion pi-ovidc direction for Ihosc seeking to improve 
the field as an academic discipline through research. 



Educational Rbsearcher^I 



The purpose of this proposed prescssion iraining session is 
lo provide methodological and non-meihodological informalion 
for those who are involved in developing and evaluation pro- 
urams or working in related research areas -.In accordance 
with the above purpose, and the need described in the introduc- 
tion, this pre session will: 

1) l-xaminc existing research methodolies which can md 
in establishing a solid discipline base for black studies. 

2) describe new and innovative methods of research which 
seem most appropriate from a black scholar s point 
of view. 

3) outline and encourage new areas of research that arc 
related to black studies ui i . ^- .k,. 

4 report on research projects involving black studies that 
arc currently in progress or that have recently been 
conipleted . . j i 

5) examine new approaches to curriculum development 
in black studies and related areas 

Anticipated Audience: The program is designed primarily for: 
n Directors of Black Studies Programs m colleges and univer- 
sities; 2) Coordinators of Black Studies Programs »n elcrncntary 
and secondary schools; 3) Research Directors and other mdivid- 
uals interested in rcscarchonblack studies. 

U. MULTIVARIATE STATISTICAL ANALYSIS IN EDU- 
CATION AL RESEARCH (Cosponsored by ETS and held 
In Princeton, NJ.) 

Director: M. I. Charles E. Woodson. Assistant Pro^^^^^^ 

School of Education, Tolman HalK University ot 
California, Berkeley, California 94720 

Gupsl Speaker: R. Gnanadesikan, Bell Telephone Laboratory . 

Staff: r/ Darren Bock, University of Chicago; Jeremy D. 

Finn, State University of New York, Buffalo and 
Ontario Institute for Educational Studies; Joel R. 
Levin, University of Wisconsin; Robert M. Pruzck. 
State University of New York, Albany; Neil 
Timm» University of Pittsburgh 

Content and Objectives: The course content will xonsist of an 
introduction to the concepts and techniques of multivariate 
analysis, including computer programs for makmg the appro- 
priate calculations. Topics will include: Univariate Analysis 
of Variance in Matrix Formulation, Hotellings T*. multivariate 
recression and canonical correlation, multivariate anaiysis ot 
variance, post-hoc tests, and other multivariate techniques, 
with an emphasis on multivariate analysis of variance. 
, The presentation will focus upon an intuitive understanding 
of multivariate procedures and their application to educational^ 
research. The mathematical development and proofs of theo- 
rems will be restricted, as much as possible, to the handout 

materials. , ... 

The primary objectives of this presession are. 

1 To prescm and interrelate the basic techniques and con- 
cepts of multivariate statistical analysis and to provide a foun- 
dation which will assist Ihe researcher in applying these to 
educational research problems. Vnnu/i 

2 To assist the participants in gaining some practical knowl- 
edge of the use of available computer programs for doing 
multivariate analysis calculations. ui ^ 

Plans have been made for participants to do problems on 
a computer during the presession. 

Anticipated Audience: This ses.sion will be open to holders of 
nocioratc in education and allied fields whose academic 
responsibilities include the design of educational research 
studies and analysis of research data. The course is intended 
for educational researchers with a strong background in quan- 
taiive methods but whose primary commitment '"ay be sub- 
Mamive areas other than statistics and experimental design. 
Participants will be expected* to have a basic knowledge ot 
algebra and the equivalent of 2 or 3 graduate courses m 
applied statistics. 

in INDIVIDUAL DIFFERENCES, LEARNING AND 
INSTRL'CiON 

nirfcctoi-i Frank H.'Fatley. University of Wisconsin, Wisconsin 
DlrectoM »• ^ anl Development Center for Cogn.iivc • 

learning. 1404 Regent Street. Madison Avenue 

5J706 

mulTi Joot L. Byeri. Michigan Stilt UnlversllVi Arthur R, 



iensen. University of Cahfornia. ^Berkeley; 
Thomas J. Shuell. State University of New York, 
Buffalo; Richard E. Snow, Stanford University 

Content and Objectives: The content of this prc^session will 
be heavily in the direction of what is presently known about 
individual differences (IDs) in learnini; and instriiction. what 
arc the best methodologies for the si^lution of the problems 
in the area, what are '.he in\pljeaUons of their knowledge anxl 
inctbodoology for educalion.il rese.\rch and what are their 
use in educational practice. Ihe seicntilic winnowini: and siU- 
inc of Ihe research evidence will be cmphasi/ed. and the 
most reccm best research will he included. The session will 
begin with basic considerations of theory and methoil. proceed 
to biological and genetic analyses, through coonccptions of m- 
trinsic IDs vs. extrinsic IDs. to the major current considera- 
tions.of aptitude x treatment interactions and programs and 
packages for individualized education. 

The objectives are primarily to provide participants with 
critical understanding of prescm-day knowledge and practice 
in the urea of individual differences and learning and give 
them the methodological and theoretical sophistication to eval- 
uate work in this areas as well as undertake research and/or 
developi^ent of their own. 

Anticipated Audience; Approximately 40-50 persons, possess- 
ing PhD degree in some area of psychology, education or 
other behavioral science, with previous university course work 
in psychology or educational psychology required. Partici- 
pants should have completed courses in the psychology of 
human learning, basic research methodology and experimental 
design or statistics. 



IV. SYSTEMS RESEARCH FOR COUNSELING AND 
EDUCATIONAL ENVIRONMENTS ; 

Director. T. A. Ryan, Researcher/Profcssor. University of 
Hawaii. Educational Research and Development 
Center, Wist Hall Annex 2. Roorn 127 1776 
University Avenue, Honolulu, Hawaii 96822 

Staff- Donald G. Hays. Union High School District. Cali- 
fornia; Leonard C. Silvern. Education and Train- 
ing Consultants. Co.; Norman R. Stewart, Michi- 
can State University; Ray E. Hosford, University 
of California. Santa Barbara; Bob B. Winborn. 
Michigan State University; James W. Lawrence, 
University of Hawaii 

Obicclives nud Content: The purpose of the presession in sys- 
tenis research is to achieve improvement in counseling, coun- 
selor education and related areas by. training counselors, conn- 
sclinc specialists, counselor educators. s\ipervisors. cducalional 
psychologists and researchers in application of systems re- 
search. It is intended that participants from urban centers will 
develop competencies which can be app led directly to re- 
.search activities which will help to J^.^^^^'^^^^S^.P^^J^^^^^^^^ 
education in the large cities. A special effort wi 1 be made to 
meet the research needs of counselors, counselor cducaiors 
supervisors and personnel in related ar?as. Three pnmafy 
uims implement the training purpose: (I) devejoping anjmj^cf. 
standing of the concepts of systems rcs^earch: 2 developing 
an understanding of the systems research; and (3) developmB 
• proficiency in using techniques of systems research A scc- 
ondary aim will he lo foster positive feelings ^-^^^^^^V "^"^^ 
concepts and techniques. The course content w 11 J^c^^ 
concepts, principles, and techtjiques of systems research ap- 
plicable in counseling and educational settings^ The coi st 
^ provide (I) instruction in basic, concepts of system 
proach. and introduction to computer simulation. (2 "sl it- 
ion relatinc to rules governing system.s research and (3) mun 
sivS traitirnV in analysis, synthesis, and. flowchart n^odcl.n,. 
and introduction to techniques of simulation. 

Anticipated Audience; The session will be open i'^ J"^'':;';|: 
u^Us in public schools, state departmenis of ^'^^^^^ " 
colleges and universities who sat.s y the following: , 
employment in a position permitting use J^y^^''^^^"^ 
in iounscling. counselor cdocat.cn. J^''^^^^^^^^ 
n) commllnint lo achieve mprovun^nl and "^novai on nuni 
W,Tc«^^^^^ and W cvlllcnce of ability lo proht from ihr 
Inniructlunul program. 



2— Educational RftafiAttCMfett 



V. KOUCATIONAL OlUF.CTIVIuS: FORMULATION, AP- 
PKAISAL, AM) ASSESSMENT 

Directors: Hva L. Uakcr. Assistant Professor. Graduate School 
—of Kducation. University of California at Los 

Anyclcs, Los Angeles. California 
W. James Popham, Professor, Graduate School of., 
l-ducaiion, University of California at Los An- 
t'L'les, Los Angeles. California 
Slair C. M. Lindvall. University of Pittsburgh; John D. 

McNeil, University of California, Los Angeles: 
kobert H. Stake, University of Illinois 

(!oiitvtil and Objectives: The prescssion is designed to promote 
thi? attainment of competency in the formulation, appraisal, 
and assessment of measurable instructional objectives. While 
cleaving to the standard AERA prescssion scheme of provid- 
ing specific skills for educational researchers, the proposed 
prescssion will in addition isolate key research and develop- 
mcnl areas wherein our insiructional objectives technology is 
paniculary deficient. *iTTe pre.session will locus on three aspects 
of educational objectives: 

Formulation: How are instructional objectives generated 
so that they will be optimally useful in instruction and evalu- 
ation settings? What are the specific techniques to be used in 
clarifying and producing such goals? 

Appraisal: How can one determine the worth of educational 
objectives, either during their formulation or after they have 
been developed? How does one establish priorities among 
competing educational objectives when designing an educa- 
tional program? 

Assessment: How can measuring procedures be developed 
for assessing the attainment of diverse types of instructional 
objectives? What types of methodological approaches should 
be used to develop defensible assessment procedures? 

Anticipated Audience: The audience for this session , will be 
both varied and large. Educational researchers in diverse pur- 
suits have found themselves faced with the need to' employ 
educational objectives in their work. Instructional designers, 
educational product developers, evaluators. to mention but a 
few, «'jll need to increase their sophistication regarding the use 
and limitations of educational objectives. 

VL NONPAUAMKTUIC MICTHODS AND RELATED 
POST HOC PROCEDURES 

Directors: Maryellen McSweeney. Michigan State University» 
464 Erickson Hall. East Lansing, Michigan 48823 
Andrew C. Porter, Michigan State University, 201 
D Erick.son Hall. East Lansing, Michigan 48823 
Staff: David J. Wright. National Assessment 

Content and Objectives: The purpose of this prescssion will 
be to increase the competency of the behavioral scientist in 
the design and analysis of experiments employing qualitative 
and quantitative variables. The prescssion will have a dual 
emphasis: 

1. the extension of contingency table techniques to com- 
plex design, testing, and estimation problems in educa- 
tional research. ^ 

2. the introduction of powerful statistical procedures which 
are less restrictive in their assumptions than are classical 
procedures and whicch are applicable to a wide variety 
of problems in data analysis. 

It is hoped that exposure to these procedures will aid the 
participant in solving problems dealing with experimental 
design* hypothesis testing, and estimation commonly found in 
educational research. 

The content of the course will be divided into three parts: 
rank procedures to test for location, contingency table pro- 
cedures for qualitative data* and techniques for measuring 
a,ssociation. To emphasize the parametric-nonparanletric an- 
alogies! we plan to begin instruction with the rank procedures. 

Anticipated Audience: This session will be open to holders 
of a doctorate or doctoral candidates in education and allied 
fields whose academic responsibilities include or will Include 
the design of educational research studies znd analysis of re- 
search data. The cour.se Is intended for educational rtstdrch- 
crs whose primary commitment Is to substantive dreas other 
than statistics and measurement. Particlt^anta Nl^ill be expected 
to have a bnsle Knowledge of Infarentlil attttistlCB, Oettcrally, 
this wilt imply a familiarity with the basic elemental^ il&tia- 
ilcDl techntquet uaiitlly preiet^tad In i (wb quiHiC or two 
scmchter courM In itatlitlci. 



VIL THE PSYCHOLOGY OF WRITTEN INSTRUCTION 

Directors; Ernest Z. Rothkopf. Bell Telephone Laboratories, 
Research and Development Unit of Bell System. 
Mountain Avenue. Murray Hill. New Jersey 
07974 

Lawrence T. Frase. Bell Telephone Laboratories. 
Research and Development Unit of Bell System. 
Mountain Avenue. Murray Hill. New Jersey 
07974 

Staff; Paul Johnson, University of Minnesota: Barbara 
Musgrave, Smith College 

Content and Objectives: The psychology of learning from 
written materials will be considered from a theoretical, experi- 
mental, and practical viewpoint. Written instruction will be 
discussed as an important current research area in educational 
psychology and as an example of problems in the ratioril 
improvement of documentary instructional systems. 
The prescssion will include four major topics. 

a. Analysis of Content. Quantitaive techniques such as di- 
graph an.nlysis, multidimensional scaling of text and of essay 
and association protocols: information analysis of content struc- 
ture. 

b. Textual Representation of Content. Syntactic and lexical 
analysis: syntactic predictors of difficulty; readability formu- 
las: computer aids in the instructional analysis of text. 

c. Psychological Models of Reading. The processes of trans- 
lating written material into internal representations, mathe- 
magenic activities; adjunct supports for study activities, use 
of questions, search procedures, and directions. 

d. Measuring the Results of Instruction. Problems in meas- 
uring text effectiveness; analysis of essay protocols and 
prompted tests, Cloze procedure, internal and external cri- 
teria for performance. 

Anticipated Audience: A doctorate in experimental psychology 
or educational psychology or equivalent experience is required. 



Xin. OPKRATIONS ANALYSLS TKCIINIOUI-^S IN EDU- 
TIONAL PLANNING AND ADMINISTRATION 

Directors: George S. Tracz. Ontario Institute for Studies in 
Education, 252 Bloor Street West, Toronto 5, 
Ontario, Canada 
James E. Bruno, Graduate School of Education, 
University of California, Los Angeles, 405 Hil- 
gard Avenue, Los Angelps, California 90024 

Staff: G. Ernest Anderson, Jr., University of Massachu- 
setts; James F. McNamara. University of Oregon 

Content and Objectives: This prescssion will expose educational 
planning personnel and school administrators to the use and 
application of management .science and operations research, 
models to problems in education. Sessions will be devoted not 
only to theory and techniques necessary for improved re- 
source allocation to and' planning of school activities, but also 
to explanations of a number of recent successful applications 
in education. Throughout the session, the utility and limitations 
of such models to improve and effect changes in educational 
planning and administrative practices will be stressed. The 
chief objectives of this prescssjon* offered for the first time, 
are these: 

(i) to provide a method of a.s.scssing the value of Opera- 
tions Analysis to operational and resource allocation 
problems in elementary and .secondary education; . 

(ii) to provide the necessary computer, and mathematical 
techniques to appreciate and understand Operations An- 
aly.sis and Systems Approaches to educational planning 
and administration; and 

(iii) to disseminate up-to-date findings in the applications of 
Operations Analy.sis to education. 

Anticipated Audience: Applications for this session arc invited 
from educational planners and administrators* from the local, 
state, and federal levels (of education) involved in the alloca* 
tion and administering of resources (Hnnncinl, fncllttlea. and 
personnel) to elementary and secondary school ncllvltles. Par- 
tlciponts then would include OSOE staff, regional laboratory 
personnel, State Education Department staff, professor;^ of ed- 
ucational administration, local school administrators, and socio- 
t^eonomfc nlanhern IHtCKBted ill the recent devalopiHent In tht 
flild of education. 



Ar'PLICATION FOR AERA RESEARCH mmiG SESSION 

DETACH AND MAIL APPLICATION TO DIRECTOR OF SESSION TO WHICH APPLYING 

GENERAL INFORMATION 



1. Session number you desire to attend— Isf choice , 2nd choice. 

2. Nome: 



3. Moiling oddrp«- '""i"! 



4. Sex: M F Age: Telephone No.: 

5. Present Institutionol Aflfiliotion (e.g., UCLA): 



6. Hove you attended on AERA Presession in the post? Yes No 
If "Yes/" when: '. and which one: 



EDUCATIONAL HISTORY 

7a. Mosters School: _ Year of Degree b. Doctoral School: Yeor of Degree 

: '- Mnjnr 

8a. Record in the blank the approximate number of COUrses you have token at either the undergraduate or graduate 
level in each of the following areas: 

o. Anthropology— ■. _ f. Linguistics. 



b. . Curriculum : ^_ , g, Mothemotics (excludir>g moth educ.) 

c. Educ. Administration ^ ^ h. Psychology (Exper., Soc, Devel., or Learning). 

d. Educ. Measurement or Psychomefrics i. Sociology 

e. Electronic Computers j. s,a,is,ics and experimental design 

b. Describe briefly your training and experience with computers: 



EMPLOYiVlENT INFORMATION 

9o. Describe briefly the nature of your present employment:. 



b. Describe briefly any changes you expect in your employment during the coming year with respect to eithe, employer 
or type of activity: 

• 10a. Whot percent of your time is allotted to teaching? b. To research? c. To grod. study? 

11. Which courses do you teach (if any), at wha» level (undergroduote-U.G.-or groduote-G), and what textbook (if any) 
might you typically use? 

C""'" Level Textbook 

: U.G r, . ' 

U.G.J G . • 

U.G G . 

U.G G : 



PROFESSIONAL AND SCHOLARLY INTERESTS 

12. Whot ore your primary research interests (e.g., moti- 14. In total, obout how mony research articles, theses or 

votion, creativity, curriculum development, experimentol technicol reports (both published and unpublished) hove 

design)? you authored alone or jointly? 



15. How rnony funded (by USOE, NIMH, Ford Foundotion, 

or o'her granting agencies) research projects ore in 

' • progress or comi3leted on which your name appears as 

either the first or a joint author? ' 

13. Approxinotcly how many research articles which you AER^Xhlt y;u';:e%'miSS 

hove outhored alone or jointly hove been accepted in 



o scholorly (refereed) journal? 



17. Describe briefly your reason for applying (use separate sheet). 

NOTE; Do not send registration fee with this application 
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INT)[<ODUCTION 



The American Educational Research Association (AERA) received a grant 
from the Office of Education to develop self-sustaining intensive training 
sessions on substantive issues in educational research. The training 
sessions were designed to cover a range of topics in educational research 
and to provide instruction that was suited to the needs of experienced 
researchers. Eight sessions were held in conjunction with AERA's 1971 
Annual Meeting in New York. A detailed final report of this project is 
included in the preceding section, 

A supplementary appropriation to the project specified above was 
awarded to AERA on June 30, 1971. A major objective of this grant was to 
design and develop additional in-service short-term training programs 
through either convention presessions (postsessions) or by other modes of 
training. These training activities were to be launched in the Spring 
or Summer of 1972. The professional stv-^ff of AERA in collaboration with 
the Association's Standing Committee on Research Training would identify 
critical areas in which in-service training was needed. More specifically, 
it would consider what training models and procedures could be produced 
to prevent potential obsolescence among educational researchers in newly 
developed methodology and technology, improve the skills and knowledge 
of persons involved in research and development roles who previously 
received only minimal training in this area, meet the needs of researchers 
outside the disciplines of education and psychology who are increasingly 
turning their attention to the study of educational issues, and for those 
involved in educational research with minorities and with the urban environment. 



ACTIVITIES 



Long-Term Evaluation 

Research Training Sessions have been sponsored by AERA since 1966* 
Evaluations of these presessions have been conducted by the session 
directors. These evaluations typically examined both the skills and 
concepts taught, and the participant's attitudes toward the learning 
experience. In recent years, the Association conducted its own evaluations 
to elicit data about the relevance of the presession topic, effectiveness 
of the instructional procedures, adequacy of the accommodations cind 
logistical support and selected demographic information. 

Commencing with the 1971 Research Training Sessions, it was decided 
to conduct a long-term follow-\:p study of possible effects of the sessions 
on the behavior of participants in such areas as their teaching, research, 
publications and consulting, - As a result of the fincincial support of this 
supplemental award, an instrument was devised and data collected to 
determine the session's influence on vrtiat trainees did when they returned 
to their home institution. The questionnaire solicited such information 
by querying participants on their pre (prior to the training session) and 
post (nine months after the session) actual research and research-related 
behavior, A detailed analysis of this data is included in the long-term 
evaluation of the Final Report of the 1971 AERA Research Training Sessions. 
Careful attention .is and will continue to be given to this report as it 
relates to future training activities. 

Experimental Session * 

The documented^^cord of success enjoyed by AERA's Research Training 
Sessions held in conjunction with the Annual Meeting suggests that replica- 



tion of this format may be the optimum model for transmitting certain 
skills and knowledge. However, the limited number of participants who 
find it possible to attend a training session that is only offered in 
the locale of the Annual Meeting, prompted a variation in this tradition. 
In this era of shrinking educational budgets, it is increasingly necessary 
to reduce the cost of instruction to individuals. Therefore, a training 
session was recently (March 20-24) conducted at Arizona State University, 
Tempe, Arizona. The 1972 Presession chairman, Thomas J. Shuell, will be 
analyzing the type of participant attending this session with the type of 
pai;ticipant attending the traditional presessions. In addition to this data, 
the Association, by virtue of this experiment, has gained valuable experience 
in the logistics and management of conducting a regional training session. 
This information will be included in the Final Report of the 1972 Research 
Training Sessions. At this point, and on the basis of only one particular 
session, the evidence suggests that it is finemcially self-supporting and 
logistically feasible for AEKA to conduct regional training sessions. 

Development of Training Models for Educational Research; A Conceptual 
Scheme for a Professional Association 

A systematic approach, working toward the solution of the complex 
problems associated with the training and retraining of research and 
research-related personnel in education, was the focus of a proposal 
(title listed above) submitted to the United States Office of Education. 
This document was a direct result of the activities specified in this 
final report and as a consequence of the Association's interest in assuming 
an active role in an ongoing training enterprise. 

Briefly stated, there are two majo): components in the project. The 
first is the creation of four alternative training formats or models, 

• 931 



4. 



a) Traveling Training Institutes, b) Intensive Pre or Post Session 
Courses, c) Annual Meeting Training Activities, and d) Instructional 
Packages. The second component seeks to develop and evaluate a conceptual 
frcunework and organizational structure, irrespective of specific personnel 
involved, by which a professional association can provide for coordinated 
and continued training activities. 

Organizational Structure 

The Association's Standing Committee on Research Training is an 
outgrowth of the 1969-71 AERA Task Force on Training Educational Research 
cuid Research-Related Personnel. The committee is composed of W. James 
Popham, University of California, Los Angeles, Chairman; Jason Millmanv 
Cornell University; Blaine Worthen, University of Colorado; Robert Morgan, 
Florida State University; Dave Merrill, Brigham Young University. Of these, 
Popham, Millman and Worthen were members of the Task Force r with Worthen 
serving as its most recent chairmcoi. 

After extensive deliberations a unique relationship between the 
specialists on AERA's Central Office staff and the representatives of the 
educational research commimity who serve on the Research Training Committee 
was agreed upon. This organizational structure, which incorporates and 
capitalizes on the expertise of the parties involved, assigns responsibilities 
as follows: The conduct of each of the four training models will be carried 
out by a member of the committee. The present assignments are: 



Model 


A 


- Traveling Training Institutes, Jim Mitchell 


Model 


B 


- Intensive Pre or Post Session Courses, Frank Farley 


Model 


C 


- Annual Meeting Tradning Activities, Jay Millman. 


Model 


b 


- Development of Instructional Packages, Dave Merrill 
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The director of each model has assumed the responsibility for its 



budget, time schedule, implementation, formative and summative evaluations. 



and final report* 



Although the individual directors of the four^mohel 



s are .given con- 



siderable latitude in the implementation and operation of their model, the 
Committee retains responsibility for overseeing the planning, designing, 
ope rationalizing, evaluating and dissemination of the objectives and/or 
products of the project. The overall coordinations of the study are the 
responsibility of the principal investigator, Richard A. Dershimer, Executive 
Officer of AERA. Administrative supervision is furnished by AERA's 
Assistant for Federal and Professional Affairs and co-director of the project, 
William J. Russell. Conceptual and substantive ^supervision is the 
responsibility of the Chairman of the Research Training Committee and co- 
director of the project, W. James Popham. Wliile the director of each 
model is charged with insuring provisions ^or.-summative and formative 
evaluations in his training activity, a member of the Committee , Blaine 
Worthen, will guide the evaluation phase of the entire project and be 
available as a consultant to the individual directors. 

Broad Questions on Research Training 



While the Committee on Research Training is integrally involved in 
the conduct of AERA's training enterprise, there is also the intention 
for them to remain sufficiently removed fraca that activity to enable them 
to consider more broadly based questions relevcmt to research training. 
In this regard, an initial and preliminsury discussion was held to generate 
possible agenda items for the Committee's consideration or review in the 
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future< In no particuleu: order, these items included: 

a) The appropriateness of rtaking formal recommendations to 

preservice training agencies regarding the nature of 
graduate tredning (e, g, , content eitqphasis, alternatives 
to qualified examination procedures, graduate student 
interchange.) 

b) Certification of individuals or training programs. 

c) Apprenticeship training, 

d) Ways to better utilize dissertation research energy potential 

for the advantage of education. 

e) Training of middle level RDD & E professionals, e.g. , 

development technicians. 

f) Appropriate interaction vehicles between the Research 

Training Committee epd the United States Office of 
Education officials responsible for research training. 

g) The appropriateness of the Committee's involvement in 

mcuipower analysis. 

h) The affects of training personnel for fields where low 

probability of employment exists. ^ 

i) Selectivity of students for training, i.e., representatives 

of minority groups as well as the identification of 
predictor variables, 
j) Other items will be submitted. 

PROGRESS 



Self-Supporting Training Sessions 

Over the years AERA's Research Training Presession progreun has 
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evolved to where It now occupies as prominent a position among the 
activities of the Association as its publications and Annual Meeting. 
After continued United States Office of Education support during its 
formative years, it ncM seems evident the sessions are a self-supporting 
activity. 

Hhis year, the following nine pre and post sessions were held: 

1. Bayeslan Statistics and Interactive Computing Systems 
Director: Melvln R. Novlck, American College Testing 
Program, Iowa. 

Staff: Nancy Cole, Axaerioan College Testing Program. 

2. Data Collection in Educational Research and Development 
Director: William E. Cbffman, College of Education, 
University of Iowa, 

Staff: Rodney Skager, University of California, Los 
Angeles I Joseph L< Mazur of South Florida. 

3. Development of Objectives-Based InBtructlonal and 
Accountability Sya terns 

Directors: Howard J.. Sullivan and Vernon derlach, 
Arizona State University. 

Staff: Fred C. Nledfirmeyer, SWRLi Richard M. Wolf, 
Columbia University. 

4. Applied Linear Regresiion Analysis in Eduoatlonal Research 
Director: Joe H. Ward, Jr., Air Foroe Human Resources 
Laboratory, San Antonio, Texas. 

Staff: Robert Bottenbergi Air Foroe Personnel Research 
Divloloni Earl Jennings, University of Texasi Janos Koplyay, 
Air Force Personal Researoh Division. 
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Instructional Objectives: Their Role in Educational 



Development and Evaluation 



Director: Eva L. Baker, Graduate School of Education, 
University of California, Los Angeles. 

Staff: W. James Popham, University of California, Los « 
Angeles; Robert E. Stcike,' University of Illinois. 

6. Research Utilization Skills for the Educational Practitioner 
Directors: Charles C« Jung, Northwest Regional Educational 
Laboratory, Portland, Oregon; Wayne Rosenoff , Far West Laboratory 
for Educational Research and Development, Berkeley, California. 
Staff: Nancy C. Adelson, Far West Laboratory for Educational 
Research and Development. 

7. Simulation Tecliniques for Evaluation Problem Solving 
Directors: Christine McGuire and Philip Bashook, Center for 
Educational Development, College of Medicine, University of 
Illinois. 

Staff: Edward Schwab, Jr., Mary Wise, Thomas Crawford, all of 
the University of Illinois. 

8. Theory and Practice of Instructional Research axxd Development 
Director: Robert D. Tennyson, Instructional DeT/elopment Center, 
Florida State University. 

Staff: Harvey Black, C. Victor Bunderson, Irwin Goodman and 
M. David Merrill, all of Brigham Young University; Robert G. 
Stakenam, Florida State University. 

9. Toward an Ethnography of Schooling 

Director: Frederick D. Erickson, Graduate School of Education, 
Harvard University. 

Staff: Stephen L. Schensul, University of Illinois; Edward T. 
Hall and Paul J. Brohemnon, Northwestern University, Stase 




McPherron, Duquesne University • 
Registration fees were assessed each participant? $100 for a three- 
day session and $150 for a five-day session. (Fourteen scholarships were 
awarded individuals from AERA funds.) This represented the Association's 
initial attempt toward making the sessions self-supporting. No outside 
funds were secured for support of these sessions this year. It is 
encouraging to note participants were willing to support such an activity. 
That is, all nine training sessions were, in fact, self -supported. A 
detailed Final Report, both financially and substantively, will be 
compiled by the chairman of the 1972 Research Training Sessions, Thomas 
Shuell. 

Traveling Training Institutes 

Substantial progress is being made toward implementation of the 
traveling research training sessions by late Summer of 1972. Outstanding 
scholars of acknowledged leadership and impact in their respective fields 
are being solicited as possible directors of sessions. Two such session 
directors have been selected to conduct regional training programs. 

One session is entitled/' Bavesian Statistics and Intera ctive Computing 
Systems , The co-directors are Melvin Novick, American College Testing 
Program and Donald Meyer, University of Pittsburgh. 

A brief description of the objectives, content and anticipated 
audience of this session follows: 

The objective of this session is to provide participants with 
a grounding in the fundamentals of Bayesian methods of statistical 
^ inference and in the employment of these methods through 
interactive computing routines. These routines are designed 
to guide the researcher who has only a minimal acquaintance 
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with Bayesian methods step-by-step through a complete Bayesian 
analysis. The interactive computing routines begin by aiding 
the researcher in interrogating himself to specify his prior 
distribution* They then perform the required calculations and 
display the posterior distribution and its important character- 
istics. 

The session will consist of a series of lectures, demonstrations 
and practicrums with considerable time being reserved for hands- 
on work at an interactive computer terminal \ander the guidance 
of a graduate assistant. A low participant to assistant ratio 
will be maintained at the terminals to assure that each parti- 
cipant receives adequate personal assistance. Terminals will 
also be made available, on a limited basis, after hours for 
participants who desire to attain a higher degree of proficiency 
in these methods or who wish to process data of their own. 
In addition to covering standard topics in the analysis of 
binominal, multinominal anoi^^Gaussian data in one or two groups, 
certain raw Bayesian Model II techniq^ies of estimating parameters 
in many groups will be described. The most interesting of 
these problems is that of multiple regression in "m" groups. 
Bayesian Model II methods for estimating means, proportions 
and regression weights in "m" groups will be discussed. 
Participants in the. training 'session can expect to obtain 
several worthwhile benefits from the training session. Among 
these are the following: 

1. Learning about some powerful new methods of data' 
analysis and their application to educational data. 
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2. Learning how interactive computing facilitates 
data analysis by the use of these methods. 

3. Learning about the use of interactive computing 
in more general educational contexts. 

Participants will be limited to forty persons. Participants 
will probably have backgrounds in the following areas: 
(a) educational measurement, (b) research design, (c) large 
scale guidance systems, (d) high school mathematics curriculum 
development, (e) t?:?st development and applications and/or 
(f) instruction in educational statistics at the undergraduate 
and graduate levels. 
The second session will be conducted by Michael Scriven, University 
of California, Berkeley and Daniel Stufflebeam, Ohio State University. 
The details of this session at this point in time are in their very formative 
stages. However, the topic of the session will be "Alternative Approaches 
To Evaluation" and will focus on rather intensive training, working from 
fairly realistic examples of evaluation problems, rather than on the abstract 
philosophy of evaluation. The intended audience is hoped to have considerable 
research expertise. 

Summary 

The Final Report of this supplementary appropriation to the 1971 
Research Training Sessions has indicated a number of activities that the 
Association was able to undertake during the past year. These activities 
included a long-term evaluation of the training .sessions, experimenting 
with a regional training session^ establishing an organizational structure 
to conduct training, and the development of other activities some of which 
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led to a proposal which provides for a systematic approach to the problem 
of training and retraining of personnel in educational research. An 
indication of progress being made in other areas is also reported. 

This report should conclude by stating the project, to a large 
extent, has bridged the'£en^oral span between the Association's formative ' 
years in providing research training with the United States Office of 
Education support to its present second generation of more sophisticated 
training activities on a self-supporting basis. Specifically, the grant 
has furnished AERA with the resources to conduct the necessary planning 
as it turns its attention to a more intensive next step toward the goal 
of a solutj^on to the problem of educational research training. 

Fiscal Report 

The final fiscal report of this supplementary grant will be submitted 
under separate cpver by the accounting office of AERA. 




